Documentation

• 1: Getting started
• 1.1: CVAT Overview
• 1.2: CVAT Complete Workflow Guide for Organizations
• 1.3: Introduction to CVAT and Datumaro
• 2: Integrations
• 2.1: FiftyOne
• 2.2: CVAT and Human Protocol
• 3: Frequently asked questions
• 4: Paid features
• 4.1: Subscription management
• 4.2: Social auth configuration
• 4.3: Shapes converter
• 4.4: Immediate job feedback
• 5: Manual
• 5.1: Basics
• 5.1.1: Registration & Account Access
• 5.1.2: Create annotation task
• 5.1.3: Create multi tasks
• 5.1.4: CVAT Annotation Interface
• 5.1.4.1: Navigation bar & Menu
• 5.1.4.2: Controls sidebar
• 5.1.4.3: Objects sidebar
• 5.1.4.4: CVAT Workspace
• 5.1.5: Jobs page
• 5.1.6: Tasks page
• 5.1.7: Task details
• 5.1.8: Requests page
• 5.1.9: 3D task workspace
• 5.1.10: Standard 3D mode (basics)
• 5.1.11: Settings
• 5.1.12: Types of shapes
• 5.1.13: Shape mode (basics)
• 5.1.14: Track mode (basics)
• 5.1.15: 3D Object annotation
• 5.1.16: Attribute annotation mode (basics)
• 5.1.17: Quality control
• 5.1.18: Vocabulary
• 5.1.19: Cloud storages page
• 5.1.20: Attach cloud storage
• 5.2: Advanced
• 5.2.1: Projects page
• 5.2.2: Organization
• 5.2.3: Search
• 5.2.4: Shape mode (advanced)
• 5.2.5: Single Shape
• 5.2.6: CVAT User roles
• 5.2.7: Track mode (advanced)
• 5.2.8: 3D Object annotation (advanced)
• 5.2.9: Attribute annotation mode (advanced)
• 5.2.10: Annotation with rectangles
• 5.2.11: Annotation with polygons
• 5.2.11.1: Manual drawing
• 5.2.11.2: Drawing using automatic borders
• 5.2.11.3: Edit polygon
• 5.2.11.4: Track mode with polygons
• 5.2.11.5: Creating masks
• 5.2.12: Annotation with polylines
• 5.2.13: Annotation with points
• 5.2.13.1: Points in shape mode
• 5.2.13.2: Linear interpolation with one point
• 5.2.14: Annotation with ellipses
• 5.2.15: Annotation with cuboids
• 5.2.15.1: Creating the cuboid
• 5.2.15.2: Editing the cuboid
• 5.2.16: Annotation with skeletons
• 5.2.17: Annotation with brush tool
• 5.2.18: Annotation with tags
• 5.2.19: Models
• 5.2.20: CVAT Analytics and QA in Cloud
• 5.2.20.1: Automated QA, Review & Honeypots
• 5.2.20.2: Manual QA and Review
• 5.2.20.3: CVAT Team Performance & Monitoring
• 5.2.21: OpenCV and AI Tools
• 5.2.22: Automatic annotation
• 5.2.23: Specification for annotators
• 5.2.24: Backup Task and Project
• 5.2.25: Frame deleting
• 5.2.26: Join and slice tools
• 5.2.27: Import datasets and upload annotation
• 5.2.28: Export annotations and data from CVAT
• 5.2.28.1: CVAT for image
• 5.2.28.2: Datumaro
• 5.2.28.3: LabelMe
• 5.2.28.4: MOT
• 5.2.28.5: MOTS
• 5.2.28.6: COCO
• 5.2.28.7: COCO Keypoints
• 5.2.28.8: Pascal VOC
• 5.2.28.9: Segmentation Mask
• 5.2.28.10: YOLO
• 5.2.28.11: YOLOv8
• 5.2.28.12: YOLOv8-Classification
• 5.2.28.13: ImageNet
• 5.2.28.14: Wider Face
• 5.2.28.15: CamVid
• 5.2.28.16: VGGFace2
• 5.2.28.17: Market-1501
• 5.2.28.18: ICDAR13/15
• 5.2.28.19: Open Images
• 5.2.28.20: Cityscapes
• 5.2.28.21: KITTI
• 5.2.28.22: LFW
• 5.2.29: XML annotation format
• 5.2.30: Shortcuts
• 5.2.31: Filter
• 5.2.32: Contextual images
• 5.2.33: Shape grouping
• 5.2.34: Dataset Manifest
• 5.2.35: Data preparation on the fly
• 5.2.36: Serverless tutorial
• 6: Administration
• 6.1: Basics
• 6.1.1: Installation Guide
• 6.1.2: Superuser registration
• 6.1.3: AWS-Deployment Guide
• 6.1.4: REST API guide
• 6.2: Advanced
• 6.2.1: CVAT Architecture
• 6.2.2: CVAT deployment on Kubernetes with Helm
• 6.2.3: Semi-automatic and Automatic Annotation
• 6.2.4: CVAT Analytics and monitoring
• 6.2.5: Mounting cloud storage
• 6.2.6: LDAP Backed Authentication
• 6.2.7: Backup guide
• 6.2.8: Upgrade guide
• 6.2.9: Webhooks
• 6.2.10: Custom Certificates
• 7: API & SDK
• 7.1: Server API
• 7.2: CVAT Python SDK
• 7.2.1: SDK API Reference
• 7.2.1.1: APIs
• 7.2.1.1.1: AnalyticsApi class reference
• 7.2.1.1.2: AssetsApi class reference
• 7.2.1.1.3: AuthApi class reference
• 7.2.1.1.4: CloudstoragesApi class reference
• 7.2.1.1.5: CommentsApi class reference
• 7.2.1.1.6: EventsApi class reference
• 7.2.1.1.7: GuidesApi class reference
• 7.2.1.1.8: InvitationsApi class reference
• 7.2.1.1.9: IssuesApi class reference
• 7.2.1.1.10: JobsApi class reference
• 7.2.1.1.11: LabelsApi class reference
• 7.2.1.1.12: LambdaApi class reference
• 7.2.1.1.13: MembershipsApi class reference
• 7.2.1.1.14: OrganizationsApi class reference
• 7.2.1.1.15: ProjectsApi class reference
• 7.2.1.1.16: QualityApi class reference
• 7.2.1.1.17: RequestsApi class reference
• 7.2.1.1.18: SchemaApi class reference
• 7.2.1.1.19: ServerApi class reference
• 7.2.1.1.20: TasksApi class reference
• 7.2.1.1.21: UsersApi class reference
• 7.2.1.1.22: WebhooksApi class reference
• 7.2.1.2: Models
• 7.2.1.2.1: About class reference
• 7.2.1.2.2: AcceptInvitationRead class reference
• 7.2.1.2.3: AnalyticsReport class reference
• 7.2.1.2.4: AnalyticsReportCreateRequest class reference
• 7.2.1.2.5: AnalyticsReportTargetEnum class reference
• 7.2.1.2.6: AnnotationConflict class reference
• 7.2.1.2.7: AnnotationConflictTypeEnum class reference
• 7.2.1.2.8: AnnotationFileRequest class reference
• 7.2.1.2.9: AnnotationGuideRead class reference
• 7.2.1.2.10: AnnotationGuideWriteRequest class reference
• 7.2.1.2.11: AnnotationId class reference
• 7.2.1.2.12: AnnotationIdShapeType class reference
• 7.2.1.2.13: AnnotationIdTypeEnum class reference
• 7.2.1.2.14: AnnotationsRead class reference
• 7.2.1.2.15: AssetRead class reference
• 7.2.1.2.16: Attribute class reference
• 7.2.1.2.17: AttributeRequest class reference
• 7.2.1.2.18: AttributeVal class reference
• 7.2.1.2.19: AttributeValRequest class reference
• 7.2.1.2.20: BackupWriteRequest class reference
• 7.2.1.2.21: BasicOrganization class reference
• 7.2.1.2.22: BasicUser class reference
• 7.2.1.2.23: BasicUserRequest class reference
• 7.2.1.2.24: BinaryOperation class reference
• 7.2.1.2.25: ChunkType class reference
• 7.2.1.2.26: ClientEvents class reference
• 7.2.1.2.27: ClientEventsRequest class reference
• 7.2.1.2.28: ClientEventsRequestPreviousEvent class reference
• 7.2.1.2.29: CloudStorageContent class reference
• 7.2.1.2.30: CloudStorageRead class reference
• 7.2.1.2.31: CloudStorageReadOwner class reference
• 7.2.1.2.32: CloudStorageWriteRequest class reference
• 7.2.1.2.33: CommentRead class reference
• 7.2.1.2.34: CommentsSummary class reference
• 7.2.1.2.35: CommentWriteRequest class reference
• 7.2.1.2.36: CredentialsTypeEnum class reference
• 7.2.1.2.37: DataFrame class reference
• 7.2.1.2.38: DataMetaRead class reference
• 7.2.1.2.39: DataRequest class reference
• 7.2.1.2.40: DataRequestValidationParams class reference
• 7.2.1.2.41: DataResponse class reference
• 7.2.1.2.42: DatasetFileRequest class reference
• 7.2.1.2.43: DatasetFormat class reference
• 7.2.1.2.44: DatasetFormats class reference
• 7.2.1.2.45: DatasetWriteRequest class reference
• 7.2.1.2.46: DefaultViewEnum class reference
• 7.2.1.2.47: Event class reference
• 7.2.1.2.48: EventRequest class reference
• 7.2.1.2.49: Events class reference
• 7.2.1.2.50: EventsEnum class reference
• 7.2.1.2.51: FileInfo class reference
• 7.2.1.2.52: FileInfoTypeEnum class reference
• 7.2.1.2.53: FrameMeta class reference
• 7.2.1.2.54: FrameSelectionMethod class reference
• 7.2.1.2.55: FunctionCall class reference
• 7.2.1.2.56: FunctionCallParams class reference
• 7.2.1.2.57: FunctionCallRequest class reference
• 7.2.1.2.58: GranularityEnum class reference
• 7.2.1.2.59: InputTypeEnum class reference
• 7.2.1.2.60: InvitationRead class reference
• 7.2.1.2.61: InvitationWriteRequest class reference
• 7.2.1.2.62: IssueRead class reference
• 7.2.1.2.63: IssuesSummary class reference
• 7.2.1.2.64: IssueWriteRequest class reference
• 7.2.1.2.65: JobAnnotationsUpdateRequest class reference
• 7.2.1.2.66: JobRead class reference
• 7.2.1.2.67: JobReadAssignee class reference
• 7.2.1.2.68: JobReadTargetStorage class reference
• 7.2.1.2.69: JobsSummary class reference
• 7.2.1.2.70: JobStage class reference
• 7.2.1.2.71: JobStatus class reference
• 7.2.1.2.72: JobType class reference
• 7.2.1.2.73: JobValidationLayoutRead class reference
• 7.2.1.2.74: JobWriteRequest class reference
• 7.2.1.2.75: Label class reference
• 7.2.1.2.76: LabeledData class reference
• 7.2.1.2.77: LabeledDataRequest class reference
• 7.2.1.2.78: LabeledImage class reference
• 7.2.1.2.79: LabeledImageRequest class reference
• 7.2.1.2.80: LabeledShape class reference
• 7.2.1.2.81: LabeledShapeRequest class reference
• 7.2.1.2.82: LabeledTrack class reference
• 7.2.1.2.83: LabeledTrackRequest class reference
• 7.2.1.2.84: LabelMappingEntryRequest class reference
• 7.2.1.2.85: LabelsSummary class reference
• 7.2.1.2.86: LocationEnum class reference
• 7.2.1.2.87: LoginSerializerExRequest class reference
• 7.2.1.2.88: MembershipRead class reference
• 7.2.1.2.89: MetaUser class reference
• 7.2.1.2.90: Metric class reference
• 7.2.1.2.91: MetricGranularity class reference
• 7.2.1.2.92: NullEnum class reference
• 7.2.1.2.93: OnlineFunctionCallRequest class reference
• 7.2.1.2.94: OperationStatus class reference
• 7.2.1.2.95: OperatorEnum class reference
• 7.2.1.2.96: OrganizationRead class reference
• 7.2.1.2.97: OrganizationWriteRequest class reference
• 7.2.1.2.98: PaginatedAnnotationConflictList class reference
• 7.2.1.2.99: PaginatedCloudStorageReadList class reference
• 7.2.1.2.100: PaginatedCommentReadList class reference
• 7.2.1.2.101: PaginatedInvitationReadList class reference
• 7.2.1.2.102: PaginatedIssueReadList class reference
• 7.2.1.2.103: PaginatedJobReadList class reference
• 7.2.1.2.104: PaginatedLabelList class reference
• 7.2.1.2.105: PaginatedMembershipReadList class reference
• 7.2.1.2.106: PaginatedMetaUserList class reference
• 7.2.1.2.107: PaginatedOrganizationReadList class reference
• 7.2.1.2.108: PaginatedProjectReadList class reference
• 7.2.1.2.109: PaginatedQualityReportList class reference
• 7.2.1.2.110: PaginatedQualitySettingsList class reference
• 7.2.1.2.111: PaginatedRequestList class reference
• 7.2.1.2.112: PaginatedTaskReadList class reference
• 7.2.1.2.113: PaginatedWebhookDeliveryReadList class reference
• 7.2.1.2.114: PaginatedWebhookReadList class reference
• 7.2.1.2.115: PasswordChangeRequest class reference
• 7.2.1.2.116: PasswordResetConfirmRequest class reference
• 7.2.1.2.117: PasswordResetSerializerExRequest class reference
• 7.2.1.2.118: PatchedAnnotationGuideWriteRequest class reference
• 7.2.1.2.119: PatchedCloudStorageWriteRequest class reference
• 7.2.1.2.120: PatchedCommentWriteRequest class reference
• 7.2.1.2.121: PatchedDataMetaWriteRequest class reference
• 7.2.1.2.122: PatchedInvitationWriteRequest class reference
• 7.2.1.2.123: PatchedIssueWriteRequest class reference
• 7.2.1.2.124: PatchedJobDataMetaWriteRequest class reference
• 7.2.1.2.125: PatchedJobValidationLayoutWriteRequest class reference
• 7.2.1.2.126: PatchedJobWriteRequest class reference
• 7.2.1.2.127: PatchedLabeledDataRequest class reference
• 7.2.1.2.128: PatchedLabelRequest class reference
• 7.2.1.2.129: PatchedMembershipWriteRequest class reference
• 7.2.1.2.130: PatchedOrganizationWriteRequest class reference
• 7.2.1.2.131: PatchedProjectWriteRequest class reference
• 7.2.1.2.132: PatchedProjectWriteRequestTargetStorage class reference
• 7.2.1.2.133: PatchedQualitySettingsRequest class reference
• 7.2.1.2.134: PatchedTaskValidationLayoutWriteRequest class reference
• 7.2.1.2.135: PatchedTaskWriteRequest class reference
• 7.2.1.2.136: PatchedTaskWriteRequestTargetStorage class reference
• 7.2.1.2.137: PatchedUserRequest class reference
• 7.2.1.2.138: PatchedWebhookWriteRequest class reference
• 7.2.1.2.139: Plugins class reference
• 7.2.1.2.140: PointSizeBaseEnum class reference
• 7.2.1.2.141: ProjectFileRequest class reference
• 7.2.1.2.142: ProjectRead class reference
• 7.2.1.2.143: ProjectReadTargetStorage class reference
• 7.2.1.2.144: ProjectWriteRequest class reference
• 7.2.1.2.145: ProviderTypeEnum class reference
• 7.2.1.2.146: QualityReport class reference
• 7.2.1.2.147: QualityReportCreateRequest class reference
• 7.2.1.2.148: QualityReportSummary class reference
• 7.2.1.2.149: QualityReportTarget class reference
• 7.2.1.2.150: QualitySettings class reference
• 7.2.1.2.151: RegisterSerializerEx class reference
• 7.2.1.2.152: RegisterSerializerExRequest class reference
• 7.2.1.2.153: Request class reference
• 7.2.1.2.154: RequestDataOperation class reference
• 7.2.1.2.155: RequestDataOperationTargetEnum class reference
• 7.2.1.2.156: RequestOwner class reference
• 7.2.1.2.157: RequestStatus class reference
• 7.2.1.2.158: RestAuthDetail class reference
• 7.2.1.2.159: RoleEnum class reference
• 7.2.1.2.160: RqId class reference
• 7.2.1.2.161: RqStatus class reference
• 7.2.1.2.162: RqStatusStateEnum class reference
• 7.2.1.2.163: SeverityEnum class reference
• 7.2.1.2.164: ShapeType class reference
• 7.2.1.2.165: SigningRequest class reference
• 7.2.1.2.166: SortingMethod class reference
• 7.2.1.2.167: Storage class reference
• 7.2.1.2.168: StorageMethod class reference
• 7.2.1.2.169: StorageRequest class reference
• 7.2.1.2.170: StorageType class reference
• 7.2.1.2.171: Sublabel class reference
• 7.2.1.2.172: SubLabeledShape class reference
• 7.2.1.2.173: SubLabeledShapeRequest class reference
• 7.2.1.2.174: SubLabeledTrack class reference
• 7.2.1.2.175: SubLabeledTrackRequest class reference
• 7.2.1.2.176: SublabelMappingEntryRequest class reference
• 7.2.1.2.177: SublabelRequest class reference
• 7.2.1.2.178: TargetMetricEnum class reference
• 7.2.1.2.179: TaskAnnotationsUpdateRequest class reference
• 7.2.1.2.180: TaskAnnotationsWriteRequest class reference
• 7.2.1.2.181: TaskFileRequest class reference
• 7.2.1.2.182: TaskRead class reference
• 7.2.1.2.183: TasksSummary class reference
• 7.2.1.2.184: TaskValidationLayoutRead class reference
• 7.2.1.2.185: TaskValidationLayoutReadMode class reference
• 7.2.1.2.186: TaskWriteRequest class reference
• 7.2.1.2.187: Token class reference
• 7.2.1.2.188: TrackedShape class reference
• 7.2.1.2.189: TrackedShapeRequest class reference
• 7.2.1.2.190: Transformation class reference
• 7.2.1.2.191: TransformationBinary class reference
• 7.2.1.2.192: User class reference
• 7.2.1.2.193: UserIdentifiers class reference
• 7.2.1.2.194: ValidationMode class reference
• 7.2.1.2.195: ValidationParamsRequest class reference
• 7.2.1.2.196: WebhookContentType class reference
• 7.2.1.2.197: WebhookDeliveryRead class reference
• 7.2.1.2.198: WebhookRead class reference
• 7.2.1.2.199: WebhookType class reference
• 7.2.1.2.200: WebhookWriteRequest class reference
• 7.2.2: Low-level API
• 7.2.3: High-level API
• 7.2.4: PyTorch adapter
• 7.2.5: Auto-annotation API
• 7.2.6: Developer guide
• 7.3: Command line interface (CLI)
• 8: Contributing to this project
• 8.1: Development environment
• 8.2: Setup additional components in development environment
• 8.3: Coding style
• 8.4: Branching model
• 8.5: Using the issue tracker
• 8.6: Bug reports
• 8.7: Feature requests
• 8.8: Pull requests
• 8.9: How to add a new annotation format support
• 8.10: Server Profiling
• 8.11: Running tests
• 8.12: Repository structure

1 - Getting started

1.1 - CVAT Overview

Machine learning systems often struggle due to poor-quality data. Without effective tools, improving a model can be tough and inefficient.

CVAT.ai is a versatile tool for annotating images and videos, serving the computer vision community worldwide.

Our goal is to help developers, businesses, and organizations globally by using a Data-centric AI approach.

CVAT offers three versions:

• CVAT Cloud: Start online with CVAT, available for free. You can also choose a subscription for unlimited data, collaboration, auto-annotations, and more.

• Self-hosted CVAT Community Edition: Follow the Self-hosted Installation Guide for setup.

• Self-hosted CVAT Enterprise Edition: We provide Enterprise-level support for this version, including premium features like SSO, LDAP, advanced integrations with Roboflow and HuggingFace, and advanced analytics. We also offer professional training and 24-hour SLA support.

See:

• Tools and formats
  • Supported formats
  • Annotation tools
• Automated labeling
• Useful links
  • Cloud
  • Self-Hosted
• Integrations
• License Information
• Get in touch

Tools and formats

CVAT stands as a comprehensive tool for image and video annotation, essential for various computer vision tasks.

It emphasizes user-friendliness, adaptability, and compatibility with a range of formats and tools.

Supported formats

CVAT’s supports the following formats:

• For 3D: .pcd, .bin
• For image: everything supported by the Python Pillow library, including formats like JPEG, PNG, BMP, GIF, PPMand TIFF.
• For video: all formats, supported by ffmpeg, including MP4, AVI, and MOV.

For annotation export and import formats, see Export annotations and data from CVAT

Annotation tools

CVAT offers a wide range of annotation tools, each catering to different aspects of image and video labeling:

These tools make CVAT a versatile platform for a range of annotation needs, from basic labeling to complex, multidimensional tasks in advanced computer vision projects.

Automated labeling

CVAT has an automated labeling features, enhancing the annotation process significantly, potentially speeding it up by up to 10 times.

Note: For more information, see OpenCV and AI Tools

Below is a detailed table of the supported algorithms and the platforms they operate on:

Useful links

Start here if you’re unsure where to begin with CVAT.

Cloud

Self-Hosted

Integrations

CVAT is a global tool, trusted and utilized by teams worldwide. Below is a list of key companies that contribute significantly to our product support or are an integral part of our ecosystem.

Note: If you’re using CVAT, we’d love to hear from you at contact@cvat.ai.

License Information

CVAT includes the following licenses:

Get in touch

To get in touch, use one of the following channels:

1.2 - CVAT Complete Workflow Guide for Organizations

Welcome to CVAT.ai, this page is the place to start your team’s annotation process using the Computer Vision Annotation Tool (CVAT).

This guide aims to equip your organization with the knowledge and best practices needed to use CVAT effectively.

We’ll walk you through every step of the CVAT workflow, from initial setup to advanced features.

See:

• Workflow diagram
• End-to-end workflow for Organizations
• Complete Workflow Guide video tutorial

Workflow diagram

The workflow diagram presents an overview of the general process at a high level.

End-to-end workflow for Organizations

To use CVAT within your organization, please follow these steps:

1. Create an account in CVAT.
2. Create Organization.
3. Switch to the Organization that you’ve created and subscribe to the Team plan.
4. Invite members to Organization and assign User roles to invited members.
5. Create Project.
6. (Optional) Attach Cloud storages to the Project.
7. Create Task or Multitask.
   At this step the CVAT platform will automatically create jobs.
8. (Optional) Create Ground truth job.
   This step can be skipped if you’re employing a manual QA approach.
9. (Optional) Add Instructions for annotators.
10. (Optional) Configure Webhooks.
11. Assign jobs to annotators by adding the annotator name to Assignee and changing the Job stage to Annotation.
12. Annotator will see assigned jobs and annotate them.
13. (Optional) In case you’ve created a Ground truth job give the CVAT platform some time to accumulate the data and check the accuracy of the annotation.
14. If you are using the manual validation, assign jobs to validators by adding the validator name to Assignee and changing the Job stage to Validation.
15. Validator will see assigned jobs and report issues.
    Note, that validators can correct issues, see Manual QA and Review
16. Check issues and if there is a need for additional improvement, reassign jobs to either the Validator or Annotator.
17. (Optional) Check Analytics.
18. Export Data.

Complete Workflow Guide video tutorial

1.3 - Introduction to CVAT and Datumaro

We are excited to introduce the first video in our course series designed to help you annotate data faster and better using CVAT. In this introductory 4 minute video, we walk through:

1. what problems CVAT and Datumaro solve,
2. how they can speed up your model training process, and
3. some resources you can use to learn more about how to use them.

2 - Integrations

2.1 - FiftyOne

FiftyOne is an open-source tool for building high-quality datasets and computer vision models. FiftyOne supercharges your machine learning workflows by enabling you to visualize datasets and interpret models faster and more effectively.

FiftyOne provides an API to create tasks and jobs, upload data, define label schemas, and download annotations using CVAT, all programmatically in Python. All of the following label types are supported, for both image and video datasets:

• Classifications
• Detections
• Instance segmentations
• Polygons and polylines
• Keypoints
• Scalar fields
• Semantic segmentation

2.2 - CVAT and Human Protocol

The integration of CVAT with HUMAN Protocol offers a groundbreaking approach to data annotation for AI and machine learning projects.

This collaboration combines CVAT’s advanced annotation tools with HUMAN Protocol’s innovative task distribution and compensation system, creating a seamless, efficient workflow for crowdsourcing annotations.

For more details on how to leverage the features of both platforms for your projects, check out these articles:

• CVAT.ai & HUMAN Protocol: A New Dawn in Visual Data Annotation
• Mastering Image Annotation Crowdsourcing for Computer Vision with CVAT.ai and HUMAN Protocol
• Crowdsourcing Annotation with CVAT and Human Protocol: Real Data Experiment Showed Amazing Results

• Basic terms
• Requester: How to get data annotated?
• Annotator: How to earn money?
• Video tutorial

Basic terms

In the realm of computer vision and AI, understanding the roles and components within data annotation projects is crucial.

Here’s a quick overview of basic terms related to CVAT and Human Protocol integration, providing a clear picture of the workflow and participants involved.

Requester: How to get data annotated?

Note: Either you are Requester or Annotator, to access this feature, you’ll need to send a request to HUMAN Protocol.

To register and launch a job on the HUMAN Protocol site, follow these steps:

1. Create a Requester account on the Human Protocol site.
2. Log in to your account and launch a Job.

The job will appear on the Annotators’ dashboard and will be annotated according to the specified requirements and quality standards.

Annotator: How to earn money?

Note: Either you are Requester or Annotator, to access this feature, you’ll need to send a request to HUMAN Protocol.

To start earning money on the Human Protocol site, follow these simple steps:

1. Register on the Human Protocol site
2. Complete KYC process.
3. Connect crypto wallet to your account.
4. Select job from Data Labeling Jobs menu.
5. Annotate.
6. After annotation is complete, change job status to Complete.

After the job is reviewed and accepted by the Requester, the money will be deposited into your account.

Video tutorial

3 - Frequently asked questions

How to migrate data from CVAT.org to CVAT.ai

Please follow the export tasks and projects guide to download an archive with data which corresponds to your task or project. The backup for a project will have all tasks which are inside the project. Thus you don’t need to export them separately.

Please follow the import tasks and projects guide to upload your backup with a task or project to a CVAT instance.

See a quick demo below. It is really a simple process. If your data is huge, it may take some time. Please be patient.

How to upgrade CVAT

Before upgrading, please follow the backup guide and backup all CVAT volumes.

Follow the upgrade guide.

How to change default CVAT hostname or port

To change the hostname, simply set the CVAT_HOST environment variable

export CVAT_HOST=<YOUR_HOSTNAME_OR_IP>

NOTE, if you’re using docker compose with sudo to run CVAT, then please add the -E (or --preserve-env) flag to preserve the user environment variable which set above to take effect in your docker containers:

sudo -E docker compose up -d

If you want to change the default web application port, change the ports part of traefik service configuration in docker-compose.yml

services:
  traefik:
    ...
    ...
    ports:
      - <YOUR_WEB_PORTAL_PORT>:8080
      - 8090:8090

Note that changing the port does not make sense if you are using HTTPS - port 443 is conventionally used for HTTPS connections, and is needed for Let’s Encrypt TLS challenge.

How to configure connected share folder on Windows

Follow the Docker manual and configure the directory that you want to use as a shared directory:

• Docker toolbox manual
• Docker for windows (see FILE SHARING section)

After that, it should be possible to use this directory as a CVAT share:

services:
  cvat_server:
    volumes:
      - cvat_share:/home/django/share:ro
  cvat_worker_import:
    volumes:
      - cvat_share:/home/django/share:ro
  cvat_worker_export:
    volumes:
      - cvat_share:/home/django/share:ro
  cvat_worker_annotation:
    volumes:
      - cvat_share:/home/django/share:ro

volumes:
  cvat_share:
    driver_opts:
      type: none
      device: /d/my_cvat_share
      o: bind

Where are uploaded images/videos stored

The uploaded data is stored in the cvat_data docker volume:

volumes:
  - cvat_data:/home/django/data

Where are annotations stored

Annotations are stored in the PostgreSQL database. The database files are stored in the cvat_db docker volume:

volumes:
  - cvat_db:/var/lib/postgresql/data

How to install CVAT on Windows 10 Home

Follow this guide.

I do not have the Analytics tab on the header section. How can I add analytics

You should build CVAT images with ‘Analytics’ component.

How to upload annotations to an entire task from UI when there are multiple jobs in the task

You can upload annotation for a multi-job task from the Dashboard view or the Task view. Uploading of annotation from the Annotation view only affects the current job.

How to specify multiple hostnames

To do this, you will need to edit traefik.http.<router>.cvat.rule docker label for both the cvat and cvat_ui services, like so (see the documentation on Traefik rules for more details):

  cvat_server:
    labels:
      - traefik.http.routers.cvat.rule=(Host(`example1.com`) || Host(`example2.com`)) &&
          PathPrefix(`/api/`, `/analytics/`, `/static/`, `/admin`, `/documentation/`, `/django-rq`)

  cvat_ui:
    labels:
      - traefik.http.routers.cvat-ui.rule=Host(`example1.com`) || Host(`example2.com`)

How to create a task with multiple jobs

Set the segment size when you create a new task, this option is available in the Advanced configuration section.

How to transfer CVAT to another machine

Follow the backup/restore guide.

How to load your own DL model into CVAT

See the information here in the Serverless tutorial.

My server uses a custom SSL certificate and I don’t want to check it.

You can call control SSL certificate check with the --insecure CLI argument. For SDK, you can specify ssl_verify = True/False in the cvat_sdk.core.client.Config object.

4 - Paid features

We provide a variety of premium features exclusively for our paying customers.

For further details, please visit:

• Tailored Cloud Plans for individuals and teams to explore paid plans on CVAT.ai.
• Self-hosted Pricing Plans to discover pricing options for community and enterprise self-hosted solutions.
• All about Pricing Plans to determine the Cloud Plan that best fits your needs.
• Advantages for Enterprise Clients to learn about the exclusive features available to Enterprise clients.

4.1 - Subscription management

This article provides tips on how to effectively manage your CVAT subscriptions, including tracking expenses and canceling unnecessary subscriptions, to optimize your finances and save time.

Whether you’re a business owner or an individual, you’ll learn how to take control of your subscriptions and manage them.

See:

• Available paid plans
  • Monthly plans
  • Annual plans
• Billing
  • How to add VAT/tax and other information to the CVAT.ai invoice before the first payment?
  • How to update VAT/tax information and other details for upcoming invoices from CVAT.ai?
  • Can a paid invoice be modified?
  • How can I get a quote before I subscribe? How to add a PO number to my invoices?
  • Can you sign an agreement before I subscribe?
  • Can you handle a bank transfer with 30-day payment terms?
  • Where can I find my invoices?
  • I am a student, can I have a discount or free access?
• Payment methods
  • Paying with bank transfer
  • How to change the payment method?
• Adding and removing team members
• Change plan
  • How to change the plan from Solo to Team?
  • How to switch from a monthly subscription to an annual one?
• Can I subscribe to several plans?
• Cancel plan
  • What will happen to my data?
  • How to cancel any plan?
  • How can I get a refund?
• Plan renewal
• Subscription management video tutorial

Available paid plans

This section outlines the paid plans available on CVAT.ai.

Monthly plans

Annual plans

Whether you’re a new user, or have a subscription to Team or Solo plan, you can subscribe to our annual plan and save up to 30% on CVAT.ai usage costs.

The annual subscription offers all the benefits of our paid plans but at a more affordable monthly rate.

For more information, see How to switch from monthly subscription to annual?

Billing

This section describes the billing model and gives short a description of limitations for each plan.

There are two types of subscriptions available for both the Solo and Team plans: monthly and annual.

For more information, see: Pricing Plans

How to add VAT/tax and other information to the CVAT.ai invoice before the first payment?

To ensure VAT (tax) information and other relevant details are included on your CVAT.ai invoices, it’s important to add this information before making the first payment.

Here’s how you can do it:

1. Sign up for a CVAT.ai account and log in.
2. (Optional) If you add the VAT/tax number to the organization, first create an organization and switch to an Organization account.
3. Navigate to the top right corner, next to the nickname, click on the arrow > upgrade to the plan.
4. Switch on the I would like the invoice to include additional data (address, phone number, VAT information) toggle, select the best payment period for you, and click Get Started.

5. You will see the billing page:

• Phone number (1).
• Billing Address: Enter the billing address you want to appear on the invoice    in the address field (2).
• VAT Information and Business Name: Select the checkbox I am purchasing as a business and enter your VAT and business name information (3).

6. Select checkbox I agree to refund policy (4).

   Note: Please read the Refund policy before selecting the checkbox.

7. Click Pay & Subscribe.

All information you’ve added will appear on the billing page and in the invoice.

By following these steps, you can seamlessly add VAT and other crucial information to your invoices, making your financial transactions with CVAT.ai transparent and compliant.

How to update VAT/tax information and other details for upcoming invoices from CVAT.ai?

In the top right corner, near the nickname, click on the arrow > manage plan.

You will see the Stripe page. Go to the Billing Information > Update Information.

Can a paid invoice be modified?

Once an invoice has been paid, it is not possible to modify it. This restriction is due to the limitations of the payment processing platform used, which in the case of CVAT.ai, is Stripe.

Stripe’s policy dictates that revisions to an invoice can only be made before payment. For more comprehensive information on this policy, please refer to Stripe’s official documentation on invoice edits at their website.

How can I get a quote before I subscribe? How to add a PO number to my invoices?

If you require a quote from CVAT for payment via bank transfer, certain criteria must be met:

• The total subscription cost must be $396 and up per year.
• Quotes are available exclusively for annual subscriptions.

Should you meet these requirements, please write to support@cvat.ai

Can you sign an agreement before I subscribe?

Sign of specific agreements and approvals are available if you meet specific criteria (the total subscription cost must be $10,000 and up per year), for more details contact support@cvat.ai

Can you handle a bank transfer with 30-day payment terms?

Yes, it is available if you fit the quota criteria, for details contact support@cvat.ai.

Where can I find my invoices?

In the top right corner, near the nickname, click on the arrow > manage plan.

You will see the Stripe page. At the bottom of the page, you will see the Invoice History section with all invoices.

Invoices are automatically sent to the account owner’s address used for the registration.

To see the invoice click on the Show Invoice Icon icon.

I am a student, can I have a discount or free access?

To consider your request for a discount, we’d need a few details from you:

• A copy of your valid student ID or any document confirming your university affiliation.
• Your university advisor’s contact details.
• The name and contact information of the dean of your faculty.
• A brief outline of your project plan. This helps us understand how we might collaborate  on a joint marketing statement highlighting your use of CVAT.ai, and how it can benefit your project.
• We’d also appreciate a positive LinkedIn post about your experience using CVAT, making sure to tag @CVAT.ai.

All these details must be sent to support@cvat.ai. Once we have this information, we’ll gladly offer you a 50% discount for one year.

Payment methods

This section describes how to change or add payment methods.

Paying with bank transfer

Note At the moment this method of payment works only with US banks.

To pay with a bank transfer:

1. Go to the Upgrade to Solo/Team plan> Get started.
2. Click US Bank Transfer.
3. Upon successful completion of the payment, you will receive a receipt via email.

Note that the completion of the payment process may take up to three banking days.

How to change the payment method?

In the top right corner, near the nickname, click on the arrow > manage plan > +Add Payment Method

Adding and removing team members

Solo plan is for personal use only, you cannot add or remove team members.

Team plan is for collaboration. To add members to your Organization, go to the Manage Team plan > Update quantity.

If you’ve added a user before the current billing period ends, the payment will be prorated for the remaining time until the next billing cycle begins. From the following month onward, the full payment will be charged.

In case you removed the user before the current billing period ends, funds will not be returned to your account, but next month you will pay less by the amount of unused funds.

Change plan

How to change the plan from Solo to Team?

The procedure is the same for both Solo and Team plans.

If for some reason you want to change your plan, you need to:

1. Unsubscribe from the previous plan.
2. If you need a refund, contact us at support@cvat.ai.
3. Subscribe to a new plan.

How to switch from a monthly subscription to an annual one?

If you have monthly subscription, and wish to switch to the Annual plan, please follow these steps:

1. In the top-right corner, near the nickname, click on the arrow.
2. Select Manage Solo/Team Plan.
3. On the Stripe page that appears, click Update Plan.

4. Choose Yearly and then click Continue.

The price will be adjusted according to the number of members, selected in the Quantity field (if updated), taking into account the amount of money that was not spent in the current period.

Upon payment, your subscription will be renewed and the start date will be reset to the day you switch to the new plan.

Can I subscribe to several plans?

Paid plans are not mutually exclusive. You can have several active subscriptions, for example, the Solo plan and several Team plans for different organizations.

Cancel plan

This section describes how to cancel your CVAT subscription and what will happen to your data.

What will happen to my data?

Once you have terminated your subscription, your data will remain accessible within the system for a month. During this period, you will be unable to add new tasks and free plan limits will be applied.

In case you possess a substantial amount of data, it will be switched to read-only mode. It means you will not be able to save annotations, add any resources, and so on.

Following the one month, you will receive a notification requesting you to either remove the excess data or it will be deleted automatically.

How to cancel any plan?

To cancel the plan, in the top right corner, near the nickname, click on the arrow> manage plan > Cancel plan

Please, fill out the feedback form, to help us improve our platform.

How can I get a refund?

To understand if you are eligible for a refund, see Refund policy.

1. Cancel the subscription before asking for a refund.
2. Contact our support team at support@cvat.ai or use the “Support” option in the app.cvat.ai interface.
3. Provide your account details and a brief explanation of the reason for the refund:
   • Send us your last invoice.
   • Send us the username and e-mail address you’ve used to register in CVAT.ai.

Our team will review your request. We may request additional information if needed. Once approved, the refund will be processed to your original payment method within 5-10 business days.

Plan renewal

To renew your CVAT.ai subscription, in the top right corner, near the nickname, click on the arrow> manage plan > Renew plan.

Subscription management video tutorial

4.2 - Social auth configuration

Note: This is a paid feature available for Enterprise clients.

You can now easily set up authentication with popular social services, which opens doors to such benefits as:

• Convenience: you can use the existing social service credentials to sign in to CVAT.
• Time-saving: with just two clicks, you can sign in without the hassle of typing in сredentials, saving time and effort.
• Security: social auth service providers have high-level security measures in place to protect your accounts.

Currently, we offer three options:

• Authentication with Github.
• Authentication with Google.
• Authentication with Amazon Cognito.

With more to come soon. Stay tuned!

See:

• Enable authentication with a Google account
• Enable authentication with a GitHub account
• Enable authentication with an Amazon Cognito

Enable authentication with a Google account

To enable authentication, do the following:

1. Log in to the Google Cloud console

2. Create a project, and go to APIs & Services

3. On the left menu, select OAuth consent, then select User type (Internal or External), and click Create.

4. On the OAuth consent screen fill all required fields, and click Save and Continue.

5. On the Scopes screen, click Add or remove scopes and select auth/userinfo.email, auth/userinfo.profile, and openid. Click Update, and Save and Continue.
   For more information, see Configure Auth Consent.

6. On the left menu, click Credentials, on the top menu click + Create credentials, and select OAuth client ID.

7. From the Application Type select Web application and configure: Application name, Authorized JavaScript origins, Authorized redirect URIs.
   For example, if you plan to deploy CVAT instance on https://localhost:8080, add https://localhost:8080 to authorized JS origins and https://localhost:8080/api/auth/social/goolge/login/callback/ to redirect URIs.

8. Create conпiguration file in CVAT:

   1. Create the auth_config.yml file with the following content:

   ---
   social_account:
     enabled: true
     google:
       client_id: <some_client_id>
       client_secret: <some_client_secret>
   

   2. Set AUTH_CONFIG_PATH="<path_to_auth_config> environment variable.

9. In a terminal, run the following command:

   docker compose -f docker-compose.yml -f docker-compose.dev.yml -f docker-compose.override.yml up -d --build
   

Enable authentication with a GitHub account

There are 2 basic steps to enable GitHub account authentication.

1. Open the GitHub settings page.

2. On the left menu, click <> Developer settings > OAuth Apps > Register new application.
   For more information, see Creating an OAuth App

3. Fill in the name field, set the homepage URL (for example: https://localhost:8080), and authentication callback URL (for example: https://localhost:8080/api/auth/social/github/login/callback/).

4. Create conпiguration file in CVAT:

   1. Create the auth_config.yml file with the following content:

   ---
   social_account:
     enabled: true
     github:
       client_id: <some_client_id>
       client_secret: <some_client_secret>
   

   2. Set AUTH_CONFIG_PATH="<path_to_auth_config> environment variable.

5. In a terminal, run the following command:

   docker compose -f docker-compose.yml -f docker-compose.dev.yml -f docker-compose.override.yml up -d --build
   

Note: You can also configure GitHub App, but don’t forget to add required permissions.
In the Permission > Account permissions > Email addresses must be set to read-only.

Enable authentication with an Amazon Cognito

To enable authentication, do the following:

1. Create a user pool. For more information, see Amazon Cognito user pools

2. Fill in the name field, set the homepage URL (for example: https://localhost:8080), and authentication callback URL (for example: https://localhost:8080/api/auth/social/amazon-cognito/login/callback/).

3. Create conпiguration file in CVAT:

   1. Create the auth_config.yml file with the following content:

   ---
   social_account:
     enabled: true
     amazon_cognito:
       client_id: <some_client_id>
       client_secret: <some_client_secret>
       domain: https://<domain-prefix>.auth.us-east-1.amazoncognito.com
   

   2. Set AUTH_CONFIG_PATH="<path_to_auth_config> environment variable.

4. In a terminal, run the following command:

   docker compose -f docker-compose.yml -f docker-compose.dev.yml -f docker-compose.override.yml up -d --build
   

4.3 - Shapes converter

The shapes converter is a feature that enables bulk actions on filtered shapes. It allows you to perform mutual conversion between masks, polygons and rectangles.

Note: All shapes converter work only when the filter is set up.

See:

• Run actions menu
• Convert shapes
• Convert shapes video tutorial

Run actions menu

Annotations actions can be accessed from the annotation menu. To access it, click on the burger icon and then select Run actions.

Note: All Shapes converter functions work in alignment with set up filter.

You will see the following dialog:

With the following fields:

Convert shapes

Recommended Precautions Before Running Annotation Actions

• Saving changes: It is recommended to save all changes prior to initiating the annotation action. If unsaved changes are detected, a prompt will advise to save these changes to avoid any potential loss of data.

• Disabу auto-save: Prior to running the annotation action, disabling the auto-save feature is advisable. A notification will suggest this action if auto-save is currently active.

• Committing changes: Changes applied during the annotation session will not be committed to the server until the saving process is manually initiated. This can be done either by the user or through the auto-save feature, should it be enabled.

To convert shapes, do the following:

1. Annotate your dataset.

   

2. Set up filters.

   

3. From the burger menu, select Run actions.

4. Choose the action you need from the Select action drop-down list.

5. (Optional) In the Starting from frame field, enter the frame number where the action should begin, and in the up to frame field, specify the frame number where the action should end.

6. (Optional) Select an option from Or choose one of the predefined options to apply the action.

7. Click Run.
   A progress bar will appear. You may abort the process by clicking Cancel until the process commits modified objects at the end of pipeline.

   

Note: Once the action is applied, it cannot be undone.

Convert shapes video tutorial

4.4 - Immediate job feedback

Overview

The basic idea behind this feature is to provide annotators with quick feedback on their performance in a job. When an annotator finishes a job, a dialog is displayed showing the quality of their annotations. The annotator can either agree or disagree with the feedback. If they disagree, they have the option to re-annotate the job and request feedback again.

To ensure transparency with the annotator, the immediate feedback shows the computed score and the minimum required score. Information about the specific errors or frames that have errors is not available to annotators.

Feedback is only available a limited number of times for each assignment, to prevent Ground Truth revealing by annotators. This is controlled by a configurable parameter, so it can be adjusted to the requirements of each project.

How to configure

Immediate feedback settings, such as Target metric, Target metric threshold, Max validations per job and others, can be configured on the quality settings page.

This feature is considered enabled if the Max validations per job is above 0. You can change the parameters any time.

Note: This feature requires a configured validation set in the task. Read more in the quality overview section or in the full guide.

1. Open the task Actions menu > Quality control > Settings

2. Set the Target metric and Target metric threshold values to what is required in your project.
3. Set Max validations per job to above zero. 3 is a good starting number.
4. Save the updated settings

How to receive a feedback

1. Assign an annotator to an annotation job
2. Annotate the job
3. Mark the job finished using the corresponding button in the menu
4. Once the job is completed, you’ll see the job validation dialog

Each assignee gets no more than the specified number of validation attempts.

Note: this functionality is only available in regular annotation jobs. For instance, it’s not possible to use it in Ground Truth jobs.

Available feedbacks

There are three types of feedbacks available for different cases:

• Accepted
• Rejected, with an option to fix mistakes
• Finally rejected when the number of attempts is exhausted

Additional details

Immediate feedback has a default timeout of 20 seconds. Feedback may be unavailable for large jobs or when there are too many immediate feedback requests. In this case annotators do not see any feedback dialogs and annotate jobs as if the feature was disabled.

The number of attempts does not decrease for staff members who have access to a job with ground truth annotations. For instance, if you’re trying to test this feature as the task owner, you may be confused if you see the number of attempts doesn’t decrease.

The number of attempts resets when the job assignee is updated.

5 - Manual

5.1 - Basics

5.1.1 - Registration & Account Access

To start to annotate in CVAT.ai, you need to create an account or log in to the existing account.

See:

• User registration
• User registration with social accounts
• Account access
• Password reset
• Change password

To create account or log in, go to the CVAT.ai Cloud login page:

Note: By default authentication and registration with Google and GitHub work only for CVAT.ai Cloud.
If you want to use Google and GitHub authentication on a local installation, see Social auth configuration.

User registration

To register, do the following:

1. Click Create an account.

   

2. Fill in all blank fields, accept terms of use, and click Create an account.

A username generates from the email automatically. You can edit it if needed.

User registration with social accounts

To register with Google or GitHub, click the button with the name of the service, and follow instructions on the screen.

Account access

To access your account, do the following:

1. Go to the login page.
2. Enter username or email. The password field will appear.
3. Enter the password and click Next.

To log in with Google or GitHub, click the button with the name of the service.

Password reset

To reset password, do the following:

1. Go to the CVAT.ai Cloud page and click Forgot password?

   

2. Enter email you used for registration and click Send.

3. Open email and click on the link from CVAT.

4. Enter new password in both fields and click Change password.

   

Change password

To change password, do the following:

1. Log in to your CVAT account.

2. In the top right corner, click on the arrow next to your nickname.

3. Select Change password.

4. Follow instructions on the screen.

   

5.1.2 - Create annotation task

To start annotating in CVAT, you need to create an annotation task and specify its parameters.

To create a task, on the Tasks page click + and select Create new task.

See:

• Create a task
  • Label shape
  • Add an attribute
  • Select files
  • Editing labels in RAW format
  • Data formats for a 3D task
• Advanced configuration

Create a task

To create a new task, open task configurator:

And specify the following parameters:

1. In the Name field, enter the name of the new task.

   

2. (Optional) From the Projects drop-down, select a project for the new task.
   Leave this field empty if you do not want to assign the task to any project.

   

   Note: Following steps are valid if the task does not belong to a project.
   If the task has been assigned to a project, the project’s labels will be applied to the task.

3. On the Constructor tab, click Add label.
   The label constructor menu will open:

   

4. In the Label name field, enter the name of the label.

5. (Optional) To limit the use of the label to a certain shape tool, from the Label shape drop-down select the shape.

6. (Optional) Select the color for the label.

   

7. (Optional) Click Add an attribute and set up its properties.

8. Click Select files to upload files for annotation.

9. Click Continue to submit the label and start adding a new one
   or Cancel to terminate the current label and return you to the labels list.

10. Click Submit and open to submit the configuration and open the created task,
    or Submit and continue, to submit the configuration and start a new task.

Label shape

Labels (or classes) are categories of objects that you can annotate.

Label shape limits the use of the label to certain shape tool.

Any is the default setting that does not limit the use of the label to any particular shape tool.

For example, you added:

• Label sun with the Label shape type ellipse
• Label car with the Label shape type any

As a result:

• The sun label will be available only for ellipse shape.

• The car label will be available for all shapes.

  

The tools on the Controls sidebar will be limited to the selected types of shapes.

For example, if you select Any, all tools will be available, but if you select Rectangle for all labels, only the Rectangle tool will be visible on the sidebar.

Note: You cannot apply the Label shape to the AI and OpenCV tools, these tools will always be available.

You can change the shape of the label as needed. This change will not affect the existing annotation.

For example, if you created objects using polygons and then changed the label shape to polylines, all previously created objects will remain polygons. However, you will not be able to add new polygon objects with the same label.

Note: You cannot change the shape of the skeleton label.
The Label shape field for the skeleton label is disabled.

Add an attribute

Attribute is a property of an annotated object, such as color, model, or other quality.

For example, you have a label for face and want to specify the type of face. Instead of creating additional labels for male and female, you can use attributes to add this information.

There are two types of attributes:

• Immutable attributes are unique and do not change from frame to frame. For example, age, gender, and color.
• Mutable attributes are temporary and can change from frame to frame. For example, pose, quality, and truncated.

Added attributes will be available from the Objects menu:

To add an attribute, do the following:

1. Go to the Constructor tab and click Add attribute.

   

2. In the Name field enter the name of the attribute.

3. From the drop-down, select way to display the attribute in the Objects menu:

   • Select enables a drop-down list, from which you can select an attribute.
     If in the Attribute value field you add __undefined__, the drop-down list will have a blank value.
     This is useful for cases where the attribute of the object cannot be clarified:

   • 

   • Radio enables the selection of one option from several options.

   • Checkbox enables the selection of multiple options.

   • Text sets the attribute to a text field.

   • Number sets the attribute to numerical field in the following format: min;max;step.

4. In the Attribute values field, add attribute values.
   To separate values use Enter.
   To delete value, use Backspace or click x next to the value name.

5. (Optional) For mutable attributes, select Mutable.

6. (Optional) To set the default attribute, hover over it with mouse cursor and click on it. The default attribute will change color to blue.

To delete an attribute, click Delete attribute.

Select files

There are several ways to upload files:

Data source	Description
My computer	Use this option to select files from your laptop or PC. To select file: 1. Click on the Select files field: . 2. Select files to upload.
Connected file share	Advanced option. Upload files from a local or cloud shared folder. Note, that you need to mount a fileshare first. For more information, see Share path
Remote source	Enter a list of URLs (one per line) in the field.
Cloud Storage	Advanced option. To upload files from cloud storage, type the cloud storage name, (optional) choose the manifest file, and select the required files. For more information, see Attach cloud storage. Use the search feature to find a file (by file name) from the connected cloud storage.

Editing labels in RAW format

The Raw is a way of working with labels for an advanced user.

It is useful when you need to copy labels from one independent task to another.

Note: Be careful with changing the raw specification of an existing task/project. Removing any “id” properties will lead to losing existing annotations. This property will be removed automatically from any text you insert to this field.

Raw presents label data in .json format with an option of editing and copying labels as text. The Done button applies the changes and the Reset button cancels the changes.

Data formats for a 3D task

To create a 3D task, you must prepare an archive with one of the following directory structures.

Note: You can’t mix 2D and 3D data in the same task.

• Velodyne
• 3D pointcloud
• 3D Option 1
• 3D Option 2

  VELODYNE FORMAT
    Structure:
      velodyne_points/
        data/
          image_01.bin
          IMAGE_00 # unknown dirname,
                   # generally image_01.png can be under IMAGE_00, IMAGE_01, IMAGE_02, IMAGE_03, etc
      data/
        image_01.png

   3D POINTCLOUD DATA FORMAT
    Structure:
      pointcloud/
        00001.pcd
      related_images/
        00001_pcd/
          image_01.png # or any other image

    3D, DEFAULT DATAFORMAT Option 1
    Structure:
      data/
        image.pcd
        image.png

    3D, DEFAULT DATAFORMAT Option 2
    Structure:
      data/
        image_1/
            image_1.pcd
            context_1.png # or any other name
            context_2.jpg

Advanced configuration

Use advanced configuration to set additional parameters for the task and customize it to meet specific needs or requirements.

The following parameters are available:

To save and open the task, click Submit & Open .

To create several tasks in sequence, click Submit & Continue.

Created tasks will be displayed on the tasks page.

5.1.3 - Create multi tasks

Use Create multi tasks to create multiple video annotation tasks with the same configuration.

To create the multi tasks, on the Tasks page click + and select Create multi tasks.

Note: The Create multi tasks feature is available for videos only.

See:

• Create multi tasks
• Example
• Errors
• Advanced configuration

Create multi tasks

To add several tasks in one go, open the task configurator:

And specify the following parameters:

1. In the Name field, enter the name of the new task:

   • Enter the name of the task
   • (Optional) {{index}} adds an index to the file in the set (starting from 0).
   • (Optional) {{file_name}} adds the file’s name to the task’s name.

2. (Optional) From the Projects drop-down, select a project for the tasks.
   Leave this field empty if you do not want to assign tasks to any project.

   

   Note: Following steps are valid if the tasks do not belong to a project.
   If the tasks have been assigned to a project, the project’s labels will be applied to the tasks.

3. On the Constructor tab, click Add label.

4. In the Label name field, enter the name of the label.

5. (Optional) Select the color for the label.

6. (Optional) Click Add an attribute and set up its properties.

7. Click Select files to upload files for annotation.

   Note: You cannot upload multiple tasks from the cloud storage.

8. Click Submit N tasks

Example

A step-by-step example for creating the multiple tasks:

1. In the Name field, enter the Create_multitask-{{index}}-{{file_name}}.

2. Add labels.

3. Select files.
   In case there are more than four files, only the total number of selected files will be displayed:

4. Click Submit N tasks

   

5. You will see a progress bar that shows the progress of the tasks being created:

   

6. Click Ok.

   

The result will look like the following:

Errors

During the process of adding multiple tasks, the following errors may occur:

Error	Description
	Wrong file format. You can add only video files.
	In the process of creating a task, CVAT was not able to process the video file. The name of the failed file will be displayed on the progress bar. To fix this issue: If you want to try again, click Retry failed tasks. If you want to skip the file, click OK.

Advanced configuration

Use advanced configuration to set additional parameters for the task and customize it to meet specific needs or requirements.

For more information, see Advanced configuration

5.1.4 - CVAT Annotation Interface

This section of the documentation describes the annotation interface and all available options that you can use to annotate image data accurately and quickly.

The interface includes the following areas:

5.1.4.1 - Navigation bar & Menu

The navigation panel and drop-down Menu, allow you to switch between frames, change the annotation mode, save your work, and more.

See:

• Menu
• Navigation bar
  • Save, Undo, Done
  • Navigation controls
• Job information and Annotation Mode switcher

Menu

Use the Menu options to upload and download annotations, change the status of the job, and access other features listed in the table below:

Panel Item	Description
Upload annotations	Upload annotations into a task.
Export as a dataset	Download a dataset in one of the supported formats.
Remove annotations	Delete all annotations for the current job. Use Select range to remove annotations for a specific range of frames. Enable the Delete only keyframe for tracks checkbox to delete only keyframes from the tracks within the selected range.
Run actions	Run annotation actions on the annotated dataset. Annotations action is a feature that allows you to modify a bulk of annotations on many frames. It supports only shape objects.
Open the task	Opens a page with details about the task.
Change job state	Changes the state of the job: New: The job is newly created and has not been started yet. It is waiting for annotation work to begin. In Progress: The job is currently being worked on. Annotations are being added or edited. Rejected: The job has been reviewed and deemed unsatisfactory or incorrect. It requires revisions and further work. Completed: The job has been finished, reviewed, and approved. No further changes are necessary.
Finish the job	Saves annotations and sets job state to Completed.

Navigation bar

Use the navigation bar to save annotation results, switch between frames, and access other features listed in the tables below.

Save, Undo, Done

Use the following buttons, to save your work, undo changes, and move tasks to done.

Function	Description
Save work	Saves annotations for the current job. The button indicates the saving process.
Undo/Redo	Use buttons to undo actions or redo them.
Done	Used to complete the creation of the object. This button appears only when the object is being created.
Block	Used to pause automatic line creation when drawing a polygon with OpenCV Intelligent scissors. Also used to postpone server requests when creating an object using AI Tools.

Navigation controls

Overview of how to navigate through frames within the interface, with detailed descriptions provided in the table below.

Function	Description
Go to the first/last frame	Navigate to the first or the last frame of the sequence.
Go back with a step/Go next with a step	Move to the previous or next frame by a predefined step. Shortcuts: C — previous frame. V — next frame. Default step size is 10 frames. To modify this, navigate to Nickname > Settings > Player Step.
Go back/Go next	Navigate to the neighboring frames. Shortcuts: D — previous frame. F — next frame. Go back/Go next buttons are customizable: To customize, right-click on the button and select one of three options (left to right): The default setting moves to the next or previous frame (step size is 1 frame). Move to the next or previous frame that contains objects (e.g., filtered). For more information, refer to Filters. Move to the next or previous frame without annotations. Use this option to quickly locate missed frames. .
Play/Pause	Switch between playing and pausing the sequence of frames or set of images. Shortcut: Space. To adjust the playback speed, go to Nickname > Settings > Player Speed.
Go to the specific frame	Enter the number of the frame you want to go to and press Enter.
Delete frame	Click to delete current frame.

Job information and Annotation Mode switcher

This section outlines various functionalities, including how to switch to the fullscreen player mode, access job information, and use the Workspace Switcher to toggle between different annotation and QA modes.

Function	Description
Fullscreen	The fullscreen player mode. The keyboard shortcut is F11.
Info	Open the job info. Overview: Assignee - the individual to whom the job is assigned. Reviewer– the user tasked with conducting the review. For more information, see Manual QA Start frame - the number of the first frame in this job. Stop frame - the number of the last frame in this job. Frames - the total number of frames in the job. Annotations Statistics table displays the number of created shapes, categorized by labels (e.g., vehicle, person) and the type of annotation (shape, track), as well as the count of manual and interpolated frames.
Filters	Switches on Filters.
Workplace Switcher	The drop-down list to swithc between different annotation modes: Overview: Standart – default mode. Attribute – annotation with Attributes Single Shape – Single shape annotation mode. Tag annotation- annotation with Tags Review – Manual QA mode.

5.1.4.2 - Controls sidebar

Navigation

Navigation block - contains tools for moving and rotating images.

Icon	Description
	Cursor (Esc)- a basic annotation pedacting tool.
	Move the image- a tool for moving around the image without the possibility of editing.
	Rotate- two buttons to rotate the current frame a clockwise (Ctrl+R) and anticlockwise (Ctrl+Shift+R). You can enable Rotate all images in the settings to rotate all the images in the job

Zoom

Zoom block - contains tools for image zoom.

Icon	Description
	Fit image- fits image into the workspace size. Shortcut - double click on an image
	Select a region of interest- zooms in on a selected region. You can use this tool to quickly zoom in on a specific part of the frame.

Shapes

Shapes block - contains all the tools for creating shapes.

Icon	Description	Links to section
	AI Tools	AI Tools
	OpenCV	OpenCV
	Rectangle	Shape mode; Track mode; Drawing by 4 points
	Polygon	Annotation with polygons; Track mode with polygons
	Polyline	Annotation with polylines
	Points	Annotation with points
	Ellipses	Annotation with ellipses
	Cuboid	Annotation with cuboids
	Brushing tools	Annotation with brushing
	Tag	Annotation with tags
	Open an issue	Review (available only in review mode)

Edit

Edit block - contains tools for editing tracks and shapes.

Icon	Description	Links to section
	Merge Shapes(M) - starts/stops the merging shapes mode.	Track mode (basics)
	Group Shapes (G) - starts/stops the grouping shapes mode.	Shape grouping
	Split - splits a track.	Track mode (advanced)
	Split - splits a track.	Track mode (advanced)
	Joins multiple labels into one	Joining mask tool
	Slices one label into several.	Slice mask/polygon

Move image

Switching between user interface modes.

1. Use arrows below to move to the next/previous frame. Use the scroll bar slider to scroll through frames. Almost every button has a shortcut. To get a hint about a shortcut, just move your mouse pointer over an UI element.

2. To navigate the image, use the button on the controls sidebar. Another way an image can be moved/shifted is by holding the left mouse button inside an area without annotated objects. If the Mouse Wheel is pressed, then all annotated objects are ignored. Otherwise the a highlighted bounding box will be moved instead of the image itself.

   

3. You can use the button on the sidebar controls to zoom on a region of interest. Use the button Fit the image to fit the image in the workspace. You can also use the mouse wheel to scale the image (the image will be zoomed relatively to your current cursor position).

   

5.1.4.3 - Objects sidebar

In the objects sidebar, you can see the list of available objects on the current frame. The following figure is an example of how the list might look like:

Shape mode	Track mode

Objects properties

Filter input box

The way how to use filters is described in the advanced guide here.

List of objects

• Switch lock property for all - switches lock property of all objects in the frame.
• Switch hidden property for all - switches hide the property of all objects in the frame.
• Expand/collapse all - collapses/expands the details field of all objects in the frame.
• Sorting - sort the list of objects: updated time, ID - accent, ID - descent

Objects on the sidebar

The type of shape can be changed by selecting the Label property. For instance, it can look like shown in the figure below:

Object action menu

The action menu calls up the button:

The action menu contains:

• Create object URL - puts a link to an object on the clipboard. After you open the link, this object will be filtered.

• Make a copy - copies an object. The keyboard shortcut is Ctrl + C > Ctrl + V.

• Propagate function copies the form to multiple frames and displays a dialog box where you can specify the number of copies or the frame to which you want to copy the object. The keyboard shortcut is Ctrl + B. On how to propagate only filtered shapes, see Shapes converter
  There are two options available:

  • Propagate forward () creates a copy of the object on N subsequent frames at the same position.
  • Propagate backward () creates a copy of the object on N previous frames at the same position.

  

• To background - moves the object to the background. The keyboard shortcut - or _

• To foreground - moves the object to the foreground. The keyboard shortcut + or =

• Change instance color- choosing a color using the color picker (available only in instance mode).

  

• Remove - removes the object. The keyboard shortcut Del, Shift+Del.

A shape can be locked to prevent its modification or moving by an accident. Shortcut to lock an object: L.

A shape can be Occluded. Shortcut: Q. Such shapes have dashed boundaries.

You can change the way an object is displayed on a frame (show or hide).

Switch pinned property - when enabled, a shape cannot be moved by dragging or dropping.

**Tracker switcher **- enable/disable tracking for the object.

By clicking on the Details button you can collapse or expand the field with all the attributes of the object.

Labels

In this tab, you can lock or hide objects of a certain label. To change the color for a specific label, you need to go to the task page and select the color by clicking the edit button, this way you will change the label color for all jobs in the task.

Fast label change

You can change the label of an object using hotkeys. In order to do it, you need to assign a number (from 0 to 9) to labels. By default numbers 1,2…0 are assigned to the first ten labels. To assign a number, click on the button placed at the right of a label name on the sidebar.

After that, you will be able to assign a corresponding label to an object by hovering your mouse cursor over it and pressing Ctrl + Num(0..9).

In case you do not point the cursor to the object, pressing Ctrl + Num(0..9) will set a chosen label as default, so that the next object you create (use the N key) will automatically have this label assigned.

Appearance

Color By options

Change the color scheme of the annotation:

• Instance — every shape has a random color

  

• Group — every group of shape has its own random color, ungrouped shapes are white

  

• Label — every label (e.g. car, person) has its own random color

  

  You can change any random color pointing to a needed box on a frame or on an object sidebar.

Fill Opacity slider

Change the opacity of every shape in the annotation.

Selected Fill Opacity slider

Change the opacity of the selected object’s fill. It is possible to change the opacity while drawing an object in the case of rectangles, polygons, and cuboids.

Outlines borders checkbox

You can change a special shape border color by clicking on the Eyedropper icon.

Show bitmap checkbox

If enabled all shapes are displayed in white and the background is black.

Show projections checkbox

Enables/disables the display of auxiliary perspective lines. Only relevant for cuboids

Hide objects sidebar

Hide - the button hides the object’s sidebar.

5.1.4.4 - CVAT Workspace

In CVAT the Workspace serves as a work area where annotators interact with images, videos, and the various tools available to create high-quality annotations.

See:

• Image settings in CVAT
  • Adding grid overlay to image in CVAT
  • Changing color settings of image in CVAT
• Adding layers and Z-axis slider
• Interacting with Objects

Image settings in CVAT

The Image settings panel serves as a versatile tool for fine-tuning the visual aspects of your image. Whether you need to brighten the image, increase contrast, or make other adjustments, this panel is your go-to.

Additionally, the panel allows you to overlay a grid on the image for more precise annotation.

Note: Adjusting the image settings only alters how the pictures are displayed. The images themselves will remain unmodified and unchanged.

By default, the Image settings panel is not visible. To access it, click on the Arrow Up () icon located at the bottom of the workspace.

Adding grid overlay to image in CVAT

To add the grid to the image, do the following:

1. Open the Image Settings panel.
2. Locate and check the box that allows you to overlay a grid on the image.
3. Specify the grid cell size in square millimeters by entering the desired number in the Size field.
4. From the Color drop-down list, select the color of the grid.
5. Use the Opacity slider to change the transparency of the grid overlay.

Changing color settings of image in CVAT

To change the color setting of the image is CVAT, do the following:

1. Open the Image Settings panel.
2. Use the slider to change the color quality.

There are four color quality settings in CVAT:

Brightness increases and decreases the overall lightness of the image:

Contrast is the range of brightness, from lightest to darkest, in an image.

Saturation describes the intensity of the color.

Gamma correction can be used to control the overall brightness of an image

To reset the setting to default values, click Reset color settings

Adding layers and Z-axis slider

Z-axis Slider enables you to add annotation layers while hiding the layers positioned beyond.

You can also move between layers by moving the slider to the layer you need.

The slider becomes active when multiple Z-layers are present within a frame. Click + on the slider to add a new layer; upon pressing it, a new layer is automatically created and activated.

You can also relocate objects between layers using the + and - keys.

Interacting with Objects

The workspace is also equipped with the following features:

• Right-clicking an object opens the Object Card. This interface contains essential controls for modifying the object’s label and attributes, as well as providing access to an action menu.

  

• Right-clicking on a polygon point will open a menu, from which you can Delete point or Set start point.

  

5.1.5 - Jobs page

On the jobs page, users (for example, with the worker role) can see the jobs that are assigned to them without having access to the task page, as well as track progress, sort and apply filters to the job list.

On the job page there is a list of jobs presented in the form of tiles, where each tile is one job. Each element contains:

• job ID
• dimension 2D or 3D
• preview
• stage and state
• when hovering over an element, you can see:
  • size
  • assignee
• menu to navigate to a task, project, or bug tracker.

To open the job in a new tab, click on the job by holding Ctrl.

In the upper left corner there is a search bar, using which you can find the job by assignee, stage, state, etc. In the upper right corner there are sorting, quick filters and filter.

Filter

Applying filter disables the quick filter.

The filter works similarly to the filters for annotation, you can create rules from properties, operators and values and group rules into groups. For more details, see the filter section. Learn more about date and time selection.

For clear all filters press Clear filters.

Supported properties for jobs list

5.1.6 - Tasks page

The tasks page contains elements and each of them relates to a separate task. They are sorted in creation order. Each element contains: task name, preview, progress bar, button Open, and menu Actions. Each button is responsible for a in menu Actions specific function:

• Export task dataset — download annotations or annotations and images in a specific format. More information is available in the export/import datasets section.
• Upload annotation upload annotations in a specific format. More information is available in the export/import datasets section.
• Automatic Annotation — automatic annotation with OpenVINO toolkit. Presence depends on how you build the CVAT instance.
• Backup task — make a backup of this task into a zip archive. Read more in the backup section.
• Move to project — Moving a task to a project (you can move only a task which does not belong to any project). In case of label mismatch, you can create or delete necessary labels in the project/task. Some task labels can be matched with the target project labels.
• Delete — delete task.

In the upper left corner there is a search bar, using which you can find the task by assignee, task name etc. In the upper right corner there are sorting, quick filters and filter.

Filter

Applying filter disables the quick filter.

The filter works similarly to the filters for annotation, you can create rules from properties, operators and values and group rules into groups. For more details, see the filter section. Learn more about date and time selection.

For clear all filters press Clear filters.

Supported properties for tasks list

Push Open button to go to task details.

5.1.7 - Task details

Task details is a task page which contains a preview, a progress bar and the details of the task (specified when the task was created) and the jobs section.

• The next actions are available on this page:

  1. Change the task’s title.
  2. Open Actions menu.
  3. Change issue tracker or open issue tracker if it is specified.
  4. Change labels (available only if the task is not related to the project). You can add new labels or add attributes for the existing labels in the Raw mode or the Constructor mode. By clicking Copy you will copy the labels to the clipboard.
  5. Assigned to — is used to assign a task to a person. Start typing an assignee’s name and/or choose the right person out of the dropdown list. In the list of users, you will only see the users of the organization where the task is created.

• Jobs — is a list of all jobs for a particular task. Here you can find the next data:

  • Jobs name with a hyperlink to it.
  • Frame range — the frame interval.
  • A stage of the job. The stage is specified by a drop-down list. There are three stages: annotation, validation or acceptance. This value affects the task progress bar.
  • A state of the job. The state can be changed by an assigned user in the menu inside the job. There are several possible states: new, in progress, rejected, completed.
  • Duration — is the amount of time the job is being worked.
  • Assignee is the user who is working on the job (annotator, reviewer or corrector). You can start typing an assignee’s name and/or choose the right person out of the dropdown list.

  You can filter or sort jobs by status, assignee and updated date using filters panel.

Follow a link inside Jobs section to start annotation process. In some cases, you can have several links. It depends on size of your task and Overlap Size and Segment Size parameters. To improve UX, only the first chunk of several frames will be loaded and you will be able to annotate first images. Other frames will be loaded in background.

5.1.8 - Requests page

The Requests page allows users to track the status of data processing jobs such as exporting annotations or importing datasets. Users can monitor progress, download results, and check for errors if they occur.

Requests List

On the Requests page, requests are displayed as cards. Each card contains the following details (if applicable):

• Operation Name
• Resource Link
• Status of the Request
• Timestamps:
  • Enqueued Date
  • Started Date
  • Finished Date
  • Result Expiration Date
• Annotations Format
• Menu to download the result or cancel a Queued job

Currently supported operations include creating tasks, importing/exporting annotations and datasets, and backups.

Statuses for Requests List

The following statuses are used to indicate the state of each request:

5.1.9 - 3D task workspace

If the related_images folder contains any images, a context image will be available in the perspective window. The contextual image could be compared to 3D data and would help to identify the labels of marked objects.

Perspective – a main window for work with objects in a 3D task.

Projections - projections are tied to an object so that a cuboid is in the center and looks like a rectangle. Projections show only the selected object.

• Top – a projection of the view from above.
• Side – a projection of the left side of the object.
• Front - a frontal projection of the object.

5.1.10 - Standard 3D mode (basics)

Standard 3d mode - Designed to work with 3D data. The mode is automatically available if you add PCD or Kitty BIN format data when you create a task. read more

You can adjust the size of the projections, to do so, simply drag the boundary between the projections.

5.1.11 - Settings

To open the settings open the user menu in the header and select the settings item or press F2.

Settings have two tabs:

In tab Player you can:

• Control step of C and V shortcuts.
• Control speed of Space/Play button.
• Select canvas background color. You can choose a background color or enter manually (in RGB or HEX format).
• Reset zoom Show every image in full size or zoomed out like previous (it is enabled by default for interpolation mode and disabled for annotation mode).
• Rotate all images checkbox — switch the rotation of all frames or an individual frame.
• Smooth image checkbox — smooth image when zoom-in it.

  smoothed	pixelized

In tab Workspace you can:

• Enable auto save checkbox — turned off by default.

• Auto save interval (min) input box — 15 minutes by default.

• Show all interpolation tracks checkbox — shows hidden objects on the side panel for every interpolated object (turned off by default).

• Always show object details - show text for an object on the canvas not only when the object is activated:

  

• Content of a text - setup of the composition of the object details:

  • ID - object identifier.
  • Attributes - attributes of the object.
  • Label - object label.
  • Source- source of creating of objects MANUAL, AUTO or SEMI-AUTO.
  • Descriptions - description of attributes.

• Position of a text - text positioning mode selection:

  • Auto - the object details will be automatically placed where free space is.
  • Center - the object details will be embedded to a corresponding object if possible.

• Font size of a text - specifies the text size of the object details.

• Automatic bordering - enable automatic bordering for polygons and polylines during drawing/editing. For more information To find out more, go to the section annotation with polygons.

• Intelligent polygon cropping - activates intelligent cropping when editing the polygon (read more in the section edit polygon

• Show tags on frame - shows/hides frame tags on current frame

• Attribute annotation mode (AAM) zoom margin input box — defines margins (in px) for shape in the attribute annotation mode.

• Control points size — defines a size of any interactable points in the tool (polygon’s vertices, rectangle dragging points, etc.)

• Default number of points in polygon approximation With this setting, you can choose the default number of points in polygon. Works for serverless interactors and OpenCV scissors.

• Click Save to save settings (settings will be saved on the server and will not change after the page is refreshed). Click Cancel or press F2 to return to the annotation.

5.1.12 - Types of shapes

There are several shapes with which you can annotate your images:

• Rectangle or Bounding box
• Polygon
• Polyline
• Points
• Ellipse
• Cuboid
• Cuboid in 3d task
• Skeleton
• Tag

And there is how they all look like:

Tag - has no shape in the workspace, but is displayed in objects sidebar.

5.1.13 - Shape mode (basics)

Usage examples:

• Create new annotations for a set of images.
• Add/modify/delete objects for existing annotations.

1. You need to select Rectangle on the controls sidebar:

   

   Before you start, select the correct Label (should be specified by you when creating the task) and Drawing Method (by 2 points or by 4 points):

   

2. Creating a new annotation in Shape mode:

   • Create a separate Rectangle by clicking on Shape.

     

   • Choose the opposite points. Your first rectangle is ready!

     

   • To learn more about creating a rectangle read here.

   • It is possible to adjust boundaries and location of the rectangle using a mouse. Rectangle’s size is shown in the top right corner , you can check it by clicking on any point of the shape. You can also undo your actions using Ctrl+Z and redo them with Shift+Ctrl+Z or Ctrl+Y.

3. You can see the Object card in the objects sidebar or open it by right-clicking on the object. You can change the attributes in the details section. You can perform basic operations or delete an object by clicking on the action menu button.

   

4. The following figure is an example of a fully annotated frame with separate shapes.

   

   Read more in the section shape mode (advanced).

5.1.14 - Track mode (basics)

Usage examples:

• Create new annotations for a sequence of frames.
• Add/modify/delete objects for existing annotations.
• Edit tracks, merge several rectangles into one track.

1. Like in the Shape mode, you need to select a Rectangle on the sidebar, in the appearing form, select the desired Label and the Drawing method.

   

2. Creating a track for an object (look at the selected car as an example):

   • Create a Rectangle in Track mode by clicking on Track.

     

   • In Track mode the rectangle will be automatically interpolated on the next frames.

   • The cyclist starts moving on frame #2270. Let’s mark the frame as a key frame. You can press K for that or click the star button (see the screenshot below).

     

   • If the object starts to change its position, you need to modify the rectangle where it happens. It isn’t necessary to change the rectangle on each frame, simply update several keyframes and the frames between them will be interpolated automatically.

   • Let’s jump 30 frames forward and adjust the boundaries of the object. See an example below:

     

   • After that the rectangle of the object will be changed automatically on frames 2270 to 2300:

     

3. When the annotated object disappears or becomes too small, you need to finish the track. You have to choose Outside Property, shortcut O.

   

4. If the object isn’t visible on a couple of frames and then appears again, you can use the Merge feature to merge several individual tracks into one.

   

   • Create tracks for moments when the cyclist is visible:

     

   • Click Merge button or press key M and click on any rectangle of the first track and on any rectangle of the second track and so on:

     

   • Click Merge button or press M to apply changes.

     

   • The final annotated sequence of frames in Interpolation mode can look like the clip below:

     

     Read more in the section track mode (advanced).

5.1.15 - 3D Object annotation

Use the 3D Annotation tool for labeling 3D objects and scenes, such as vehicles, buildings, landscapes, and others.

See:

• Navigation
• Annotation with cuboids
  • Annotation with shapes
  • Tracking with cuboids

Navigation

The 3D annotation canvas looks like the following

Note: if you added contextual images to the dataset, the canvas will include them. For more information, see Contextual images

For information on the available tools, see Controls sidebar.

You can navigate, using the mouse, or navigation keys:

You can also use keyboard shortcuts to navigate:

Annotation with cuboids

There are two options available for 3D annotation:

• Shape: for tasks like object detection.
• Track: uses interpolation to predict the position of objects in subsequent frames. A unique ID will be assigned to each object and maintained throughout the sequence of images.

Annotation with shapes

To add a 3D shape, do the following:

1. On the objects pane, select Draw new cuboid > select the label from the drop-down list > Shape.

2. The cursor will be followed by a cuboid. Place the cuboid on the 3D scene.

3. Use projections to adjust the cuboid. Click and hold the left mouse button to edit the label shape on the projection.

4. (Optional) Move one of the four points to change the size of the cuboid.

5. (Optional) To rotate the cuboid, click on the middle point and then drag the cuboid up/down or to left/right.

   

Tracking with cuboids

To track with cuboids, do the following:

1. On the objects pane, select Draw new cuboid > select the label from the drop-down list > Track.

2. The cursor will be followed by a cuboid. Place the cuboid on the 3D scene.

3. Use projections to adjust the cuboid. Click and hold the left mouse button to edit the label shape on the projection.

4. (Optional) Move one of the four points to change the size of the cuboid.

5. (Optional) To rotate the cuboid, click on the middle point and then drag the cuboid up/down or to left/right.

6. Move several frames forward. You will see the cuboid you’ve added in frame 1. Adjust it, if needed.

7. Repeat to the last frame with the presence of the object you are tracking.

For more information about tracking, see Track mode

5.1.16 - Attribute annotation mode (basics)

• In this mode you can edit attributes with fast navigation between objects and frames using a keyboard. Open the drop-down list in the top panel and select Attribute annotation Mode.

  

• In this mode objects panel change to a special panel :

  

• The active attribute will be red. In this case it is gender . Look at the bottom side panel to see all possible shortcuts for changing the attribute. Press key 2 on your keyboard to assign a value (female) for the attribute or select from the drop-down list.

  

• Press Up Arrow/Down Arrow on your keyboard or click the buttons in the UI to go to the next/previous attribute. In this case, after pressing Down Arrow you will be able to edit the Age attribute.

  

• Use Right Arrow/Left Arrow keys to move to the previous/next image with annotation.

To see all the hot keys available in the attribute annotation mode, press F2. Read more in the section attribute annotation mode (advanced).

5.1.17 - Quality control

CVAT has the following features for automated quality control of annotations:

• Validation set configuration for a task
• Job validation on job finish (“Immediate feedback”)
• Review mode for problems found
• Quality analytics

In this section we only highlight the key steps in quality estimation. Read the detailed guide on quality estimation in CVAT in the Advanced section.

How to enable quality control

• In a new task
• In an existing task

How to enable immediate job feedback

Note: This feature requires a configured validation set in the task. Read more in How to enable quality control and in the full guide.

1. Open the task Actions menu > Quality control > Settings
2. Set Max validations per job to above zero. 3 is a good starting number.

3. Save the updated settings
4. Assign an annotator to an annotation job
5. Annotate the job
6. Mark the job finished using the corresponding button in the menu
7. Once the job is completed, you’ll see the job validation dialog

Each assignee gets no more than the specified number of validation attempts.

Read more about this functionality in the Immediate Feedback section.

How to check task quality metrics

1. Open the task Actions menu > Quality control
2. (optional) Request quality metrics computation, wait for completion
3. Review summaries or detailed reports

Read more about this functionality here.

How to review problems found

1. Open the task Actions menu > Quality control
2. Find an annotation job to be reviewed, it must have at least 1 validation frame
3. Click the job link
4. Switch to the Review mode
5. Enable display of Ground Truth annotations and conflicts

Read more about this functionality here.

5.1.18 - Vocabulary

Label

Label is a type of an annotated object (e.g. person, car, vehicle, etc.)

Attribute

Attribute is a property of an annotated object (e.g. color, model, quality, etc.). There are two types of attributes:

Unique

Unique immutable and can’t be changed from frame to frame (e.g. age, gender, color, etc.)

Temporary

Temporary mutable and can be changed on any frame (e.g. quality, pose, truncated, etc.)

Track

Track is a set of shapes on different frames which corresponds to one object. Tracks are created in Track mode

Annotation

Annotation is a set of shapes and tracks. There are several types of annotations:

• Manual which is created by a person
• Semi-automatic which is created mainly automatically, but the user provides some data (e.g. interpolation)
• Automatic which is created automatically without a person in the loop

Approximation

Approximation allows you to reduce the number of points in the polygon. Can be used to reduce the annotation file and to facilitate editing polygons.

Trackable

Trackable object will be tracked automatically if the previous frame was a latest keyframe for the object. More details in the section trackers.

Mode

Interpolation

Mode for video annotation, which uses track objects. Only objects on keyframes are manually annotation, and intermediate frames are linearly interpolated.

Related sections:

• Track mode

Annotation

Mode for images annotation, which uses shape objects.

Related sections:

• Shape mode

Dimension

Depends on the task data type that is defined when the task is created.

2D

The data format of 2d tasks are images and videos. Related sections:

• Creating an annotation task

3D

The data format of 3d tasks is a cloud of points. Data formats for a 3D task

Related sections:

• 3D task workspace
• Standard 3D mode
• 3D Object annotation

State

State of the job. The state can be changed by an assigned user in the menu inside the job. There are several possible states: new, in progress, rejected, completed.

Stage

Stage of the job. The stage is specified with the drop-down list on the task page. There are three stages: annotation, validation or acceptance. This value affects the task progress bar.

Subset

A project can have subsets. Subsets are groups for tasks that make it easier to work with the dataset. It could be test, train, validation or custom subset.

Credentials

Under credentials is understood Key & secret key, Account name and token, Anonymous access, Key file. Used to attach cloud storage.

Resource

Under resource is understood bucket name or container name. Used to attach cloud storage.

5.1.19 - Cloud storages page

The cloud storages page contains elements, each of them relating to a separate cloud storage.  Each element contains: preview, cloud storage name, provider, creation and update info, status, ? button for displaying the description and the actions menu.

Each button in the action menu is responsible for a specific function:

• Update — update this cloud storage
• Delete — delete cloud storage.

This preview will appear when it is impossible to get a real preview (e.g. storage is empty or invalid credentials were used).

In the upper left corner there is a search bar, using which you can find the cloud storage by display name, provider, etc. In the upper right corner there are sorting, quick filters and filter.

Filter

Applying filter disables the quick filter.

The filter works similarly to the filters for annotation, you can create rules from properties, operators and values and group rules into groups. For more details, see the filter section. Learn more about date and time selection.

For clear all filters press Clear filters.

Supported properties for cloud storages list

Click the + button to attach a new cloud storage.

5.1.20 - Attach cloud storage

In CVAT you can use AWS S3, Azure Blob Storage and Google Cloud Storage storages to import and export image datasets for your tasks.

See:

• AWS S3
  • Create a bucket
  • Upload data
  • Access permissions
    • Authenticated access
    • Anonymous access
  • Attach AWS S3 storage
  • AWS S3 manifest file
  • Video tutorial: Add AWS S3 as Cloud Storage in CVAT
• Google Cloud Storage
  • Create a bucket
  • Upload data
  • Access permissions
    • Authenticated access
    • Anonymous access
  • Attach Google Cloud Storage
  • Video tutorial: Add Google Cloud Storage as Cloud Storage in CVAT
• Microsoft Azure Blob Storage
  • Create a bucket
  • Create a container
  • Upload data
  • SAS token and connection string
  • Personal use
  • Attach Azure Blob Storage
  • Video tutorial: Add Microsoft Azure Blob Storage as Cloud Storage in CVAT
• Prepare the dataset

AWS S3

Create a bucket

To create bucket, do the following:

1. Create an AWS account.

2. Go to console AWS-S3, and click Create bucket.

   

3. Specify the name and region of the bucket. You can also copy the settings of another bucket by clicking on the Choose bucket button.

4. Enable Block all public access. For access, you will use access key ID and secret access key.

5. Click Create bucket.

A new bucket will appear on the list of buckets.

Upload data

Note: manifest file is optional.

You need to upload data for annotation and the manifest.jsonl file.

1. Prepare data. For more information, see prepare the dataset.

2. Open the bucket and click Upload.

   

3. Drag the manifest file and image folder on the page and click Upload:

Access permissions

Authenticated access

To add access permissions, do the following:

1. Go to IAM and click Add users.

2. Set User name and enable Access key - programmatic access.

   

3. Click Next: Permissions.

4. Click Create group, enter the group name.

5. Use search to find and select:

   • For read-only access: AmazonS3ReadOnlyAccess.
   • For full access: AmazonS3FullAccess.

   

6. (Optional) Add tags for the user and go to the next page.

7. Save Access key ID and Secret access key.

For more information, see Creating an IAM user in your AWS account

Anonymous access

On how to grant public access to the bucket, see Configuring block public access settings for your S3 buckets

Attach AWS S3 storage

To attach storage, do the following:

1. Log into CVAT and in the separate tab open your bucket page.
2. In the CVAT, on the top menu select Cloud storages > on the opened page click +.

Fill in the following fields:

After filling in all the fields, click Submit.

AWS S3 manifest file

Note: manifest file is optional.

To prepare the manifest file, do the following:

1. Go to AWS CLI and run script for prepare manifest file.
2. Perform the installation, following the aws-shell manual,
   You can configure credentials by running aws configure.
   You will need to enter Access Key ID and Secret Access Key as well as the region.

aws configure
Access Key ID: <your Access Key ID>
Secret Access Key: <your Secret Access Key>

3. Copy the content of the bucket to a folder on your computer:

aws s3 cp <s3://bucket-name> <yourfolder> --recursive

4. After copying the files, you can create a manifest file as described in preapair manifest file section:

python <cvat repository>/utils/dataset_manifest/create.py --output-dir <yourfolder> <yourfolder>

5. When the manifest file is ready, upload it to aws s3 bucket:

• For read and write permissions when you created the user, run:

aws s3 cp <yourfolder>/manifest.jsonl <s3://bucket-name>

• For read-only permissions, use the download through the browser, click upload, drag the manifest file to the page and click upload.

Video tutorial: Add AWS S3 as Cloud Storage in CVAT

Google Cloud Storage

Create a bucket

To create bucket, do the following:

1. Create Google account and log into it.
2. On the Google Cloud page, click Start Free, then enter the required data and accept the terms of service.

   Note: Google requires to add payment, you will need a bank card to accomplish step 2.

3. Create a Bucket with the following parameters:
   • Name your bucket: Unique name.
   • Choose where to store your data: Set up a location nearest to you.
   • Choose a storage class for your data: Set a default class > Standard.
   • Choose how to control access to objects: Enforce public access prevention on this bucket > Uniform (default).
   • How to protect data: None

You will be forwarded to the bucket.

Upload data

Note: manifest file is optional.

You need to upload data for annotation and the manifest.jsonl file.

1. Prepare data. For more information, see prepare the dataset.
2. Open the bucket and from the top menu select Upload files or Upload folder (depends on how your files are organized).

Access permissions

To access Google Cloud Storage get a Project ID from cloud resource manager page

And follow instructions below based on the preferable type of access.

Authenticated access

For authenticated access you need to create a service account and key file.

To create a service account:

1. On the Google Cloud platform, go to IAM & Admin > Service Accounts and click +Create Service Account.
2. Enter your account name and click Create And Continue.
3. Select a role, for example Basic > Viewer, and click Continue.
4. (Optional) Give access rights to the service account.
5. Click Done.

To create a key:

1. Go to IAM & Admin > Service Accounts > click on account name > Keys.
2. Click Add key and select Create new key > JSON
3. Click Create. The key file will be downloaded automatically.

For more information about keys, see Learn more about creating keys.

Anonymous access

To configure anonymous access:

1. Open the bucket and go to the Permissions tab.
2. Сlick + Grant access to add new principals.
3. In the New principals field specify allUsers, select roles: Cloud Storage Legacy > Storage Legacy Bucket Reader.
4. Click Save.

Now you can attach the Google Cloud Storage bucket to CVAT.

Attach Google Cloud Storage

To attach storage, do the following:

1. Log into CVAT and in the separate tab open your bucket page.
2. In the CVAT, on the top menu select Cloud storages > on the opened page click +.

Fill in the following fields:

After filling in all the fields, click Submit.

Video tutorial: Add Google Cloud Storage as Cloud Storage in CVAT

Microsoft Azure Blob Storage

Create a bucket

To create bucket, do the following:

1. Create an Microsoft Azure account and log into it.

2. Go to Azure portal, hover over the resource , and in the pop-up window click Create.

   

3. Enter a name for the group and click Review + create, check the entered data and click Create.

4. Go to the resource groups page, navigate to the group that you created and click Create resources.

5. On the marketplace page, use search to find Storage account.

   

6. Click on Storage account and on the next page click Create.

7. On the Basics tab, fill in the following fields:

   • Storage account name: to access container from CVAT.
   • Select a region closest to you.
   • Select Performance > Standart.
   • Select Local-redundancy storage (LRS).
   • Click next: Advanced>.

   

8. On the Advanced page, fill in the following fields:

   • (Optional) Disable Allow enabling public access on containers to prohibit anonymous access to the container.
   • Click Next > Networking.

9. On the Networking tab, fill in the following fields:

   • If you want to change public access, enable Public access from all networks.

   • Click Next>Data protection.

     You do not need to change anything in other tabs until you need some specific setup.

10. Click Review and wait for the data to load.

11. Click Create. Deployment will start.

12. After deployment is over, click Go to resource.

Create a container

To create container, do the following:

1. Go to the containers section and on the top menu click +Container

3. Enter the name of the container.
4. (Optional) In the Public access level drop-down, select type of the access.
   Note: this field will inactive if you disabled Allow enabling public access on containers.
5. Click Create.

Upload data

You need to upload data for annotation and the manifest.jsonl file.

1. Prepare data. For more information, see prepare the dataset.
2. Go to container and click Upload.
3. Click Browse for files and select images.

   Note: If images are in folder, specify folder in the Advanced settings > Upload to folder.

4. Click Upload.

SAS token and connection string

Use the SAS token or connection string to grant secure access to the container.

To configure the credentials:

1. Go to Home > Resourse groups > You resource name > Your storage account.
2. On the left menu, click Shared access signature.
3. Change the following fields:
   • Allowed services: Enable Blob . Disable all other fields.
   • Allowed resource types: Enable Container and Object. Disable all other fields.
   • Allowed permissions: Enable Read, Write, and List. Disable all other fields.
   • Start and expiry date: Set up start and expiry dates.
   • Allowed protocols: Select HTTPS and HTTP
   • Leave all other fields with default parameters.
4. Click Generate SAS and connection string and copy SAS token or Connection string.

Personal use

For personal use, you can use the Access Key from your storage account in the CVAT SAS Token field.

To get the Access Key:

1. In the Azure Portal, go to the Security + networking > Access Keys
2. Click Show and copy the key.

Attach Azure Blob Storage

To attach storage, do the following:

1. Log into CVAT and in the separate tab open your bucket page.
2. In the CVAT, on the top menu select Cloud storages > on the opened page click +.

Fill in the following fields:

After filling in all the fields, click Submit.

Video tutorial: Add Microsoft Azure Blob Storage as Cloud Storage in CVAT

Prepare the dataset

For example, the dataset is The Oxford-IIIT Pet Dataset:

1. Download the archive with images.
2. Unpack the archive into the prepared folder.
3. Create a manifest. For more information, see Dataset manifest:

python <cvat repository>/utils/dataset_manifest/create.py --output-dir <your_folder> <your_folder>

5.2 - Advanced

5.2.1 - Projects page

Projects page

On this page you can create a new project, create a project from a backup, and also see the created projects.

In the upper left corner there is a search bar, using which you can find the project by project name, assignee etc. In the upper right corner there are sorting, quick filters and filter.

Filter

Applying filter disables the quick filter.

The filter works similarly to the filters for annotation, you can create rules from properties, operators and values and group rules into groups. For more details, see the filter section. Learn more about date and time selection.

For clear all filters press Clear filters.

Supported properties for projects list

Create a project

At CVAT, you can create a project containing tasks of the same type. All tasks related to the project will inherit a list of labels.

To create a project, go to the projects section by clicking on the Projects item in the top menu. On the projects page, you can see a list of projects, use a search, or create a new project by clicking on the + button and select Create New Project.

Note that the project will be created in the organization that you selected at the time of creation. Read more about organizations.

You can change: the name of the project, the list of labels (which will be used for tasks created as parts of this project) and a skeleton if it’s necessary. In advanced configuration also you can specify: a link to the issue, source and target storages. Learn more about creating a label list, creating the skeleton and attach cloud storage.

To save and open project click on Submit & Open button. Also you can click on Submit & Continue button for creating several projects in sequence

Once created, the project will appear on the projects page. To open a project, just click on it.

Here you can do the following:

1. Change the project’s title.

2. Open the Actions menu. Each button is responsible for a specific function in the Actions menu:

   • Export dataset/Import dataset - download/upload annotations or annotations and images in a specific format. More information is available in the export/import datasets section.
   • Backup project - make a backup of the project read more in the backup section.
   • Delete - remove the project and all related tasks.

3. Change issue tracker or open issue tracker if it is specified.

4. Change labels and skeleton. You can add new labels or add attributes for the existing labels in the Raw mode or the Constructor mode. You can also change the color for different labels. By clicking Setup skeleton you can create a skeleton for this project.

5. Assigned to — is used to assign a project to a person. Start typing an assignee’s name and/or choose the right person out of the dropdown list.

6. Tasks — is a list of all tasks for a particular project, with the ability to search, sort and filter for tasks in the project. Read more about search. Read more about sorting and filter It is possible to choose a subset for tasks in the project. You can use the available options (Train, Test, Validation) or set your own.

5.2.2 - Organization

Organization is a feature for teams of several users who work together on projects and share tasks.

Create an Organization, invite your team members, and assign roles to make the team work better on shared tasks.

See:

• Personal workspace
• Create new organization
  • Switching between organizations
• Organization page
• Invite members into organization: menu and roles
  • Inviting members to Organization
  • Invitations list
  • Resending and removing invitations
• Delete organization

Personal workspace

The account’s default state is activated when no Organization is selected.

If you do not select an Organization, the system links all new resources directly to your personal account, that inhibits resource sharing with others.

When Personal workspace is selected, it will be marked with a tick in the menu.

Create new organization

To create an organization, do the following:

1. Log in to the CVAT.

2. On the top menu, click your Username > Organization > + Create.

   

3. Fill in the following fields and click Submit.

   

Upon creation, the organization page will open automatically.

For future access to your organization, navigate to Username > Organization

Note, that if you’ve created more than 10 organizations, a Switch organization line will appear in the drop-down menu.

Switching between organizations

If you have more than one Organization, it is possible to switch between these Organizations at any given time.

Follow these steps:

1. In the top menu, select your Username > Organization.
2. From the drop-down menu, under the Personal space section, choose the desired Organization.

Note, that if you’ve created more than 10 organizations, a Switch organization line will appear in the drop-down menu.

Click on it to see the Select organization dialog, and select organization from drop-down list.

Organization page

Organization page is a place, where you can edit the Organization information and manage Organization members.

Note that in order to access the organization page, you must first activate the organization (see Switching between organizations). Without activation, the organization page will remain inaccessible.
An organization is considered activated when it’s ticked in the drop-down menu and its name is visible in the top-right corner under the username.

To go to the Organization page, do the following:

1. On the top menu, click your Username > Organization.
2. In the drop-down menu, select Organization.
3. In the drop-down menu, click Settings.

Invite members into organization: menu and roles

Invite members form is available from Organization page.

It has the following fields:

Members of Organization will appear on the Organization page:

The member of the organization can leave the organization by going to Organization page > Leave organization.

Inviting members to Organization

To invite members to Organization do the following:

1. Go to the Organization page, and click Invite members.

2. Fill in the form (see below).

   

3. Click Ok.

4. The person being invited will receive an email with the link.

   

5. Person must click the link and:

   1. If the invitee does not have the CVAT account, then set up an account.
   2. If the invitee has a CVAT account, then log in to the account.

Invitations list

User can see the list of active invitations.

To see the list, Go to Username > Organization > Invitations.

You will see the page with the list of invitations.

You will also see pop-up notification the link to the page with invitations list.

Resending and removing invitations

The organization owner and maintainers can remove members, by clicking on the three dots, and selecting Remove invitation

The organization owner can remove members, by clicking on the Bin icon.

Delete organization

You can remove an organization that you created.

Note: Removing an organization will delete all related resources (annotations, jobs, tasks, projects, cloud storage, and so on).

To remove an organization, do the following:

1. Go to the Organization page.
2. In the top-right corner click Actions > Remove organization.
3. Enter the short name of the organization in the dialog field.
4. Click Remove.

5.2.3 - Search

There are several options how to use the search.

• Search within all fields (owner, assignee, task name, task status, task mode). To execute enter a search string in search field.
• Search for specific fields. How to perform:
  • owner: admin - all tasks created by the user who has the substring “admin” in his name
  • assignee: employee - all tasks which are assigned to a user who has the substring “employee” in his name
  • name: training - all tasks with the substring “training” in their names
  • mode: annotation or mode: interpolation - all tasks with images or videos.
  • status: annotation or status: validation or status: completed - search by status
  • id: 5 - task with id = 5.
• Multiple filters. Filters can be combined (except for the identifier) ​​using the keyword AND:
  • mode: interpolation AND owner: admin
  • mode: annotation and status: annotation

The search is case insensitive.

5.2.4 - Shape mode (advanced)

Basic operations in the mode were described in section shape mode (basics).

Occluded Occlusion is an attribute used if an object is occluded by another object or isn’t fully visible on the frame. Use Q shortcut to set the property quickly.

Example: the three cars on the figure below should be labeled as occluded.

If a frame contains too many objects and it is difficult to annotate them due to many shapes placed mostly in the same place, it makes sense to lock them. Shapes for locked objects are transparent, and it is easy to annotate new objects. Besides, you can’t change previously annotated objects by accident. Shortcut: L.

5.2.5 - Single Shape

The CVAT Single Shape annotation mode accelerates the annotation process and enhances workflow efficiency for specific scenarios.

By using this mode you can label objects with a chosen annotation shape and label when an image contains only a single object. By eliminating the necessity to select tools from the sidebar and facilitating quicker navigation between images without the reliance on hotkeys, this feature makes the annotation process significantly faster.

See:

• Single Shape mode annotation interface
• Annotating in Single Shape mode
• Query parameters
• Video tutorial

Single Shape mode annotation interface

A set of controls in the interface of the Single Shape annotation mode may vary depends on different settings.

Images below displays the complete interface, featuring all available fields; as mentioned above, certain fields may be absent depending on the scenario.

For instance, when annotating with rectangles, the Number of points field will not appear, and if annotating a single class, the Labels selector will be omitted.

To access Single Shape mode, open the job, navigate to the top right corner, and from the drop-down menu, select Single Shape.

The interface will be different if the shape type was set to Any in the label Constructor:

The Single Shape annotation mode has the following fields:

Annotating in Single Shape mode

To annotate in Single Shape mode, follow these steps:

1. Open the job and switch to Single Shape mode.
2. Annotate the image based on the selected shape. For more information on shapes, see Annotation Tools.
3. (Optional) If the image does not contain any objects to annotate, click Skip at the top of the right panel.
4. Submit your work.

Query parameters

Also, we introduced additional query parameters, which you may append to the job link, to initialize the annotation process and automate workflow:

You can combine these parameters to customize the workspace for an annotator, for example:

/tasks/<tid>/jobs/<jid>?defaultWorkspace=single_shape&defaultLabel=dog&defaultPointsCount=10

Will open the following job:

Video tutorial

For a better understanding of how Single Shape mode operates, we recommend watching the following tutorial.

5.2.6 - CVAT User roles

CVAT offers two distinct types of roles:

• Global Roles: These are universal roles that apply to the entire system. Anyone who logs into the CVAT.ai platform is automatically assigned a global role. It sets the basic permissions that every registered user has across CVAT.ai, regardless of their specific tasks or responsibilities.
• Organization Roles: These roles determine what a user can do within the Organization, allowing for more tailored access based on the user’s specific duties and responsibilities.

Organization roles complement global roles by determining the visibility of different resources for example, tasks or jobs.

Limits: Limits are applicable to all users of the CVAT.ai Cloud Platform using the Free plan and can be lifted upon choosing a subscription.

All roles are predefined and cannot be modified through the user interface. However, within the self-hosted solution, roles can be adjusted using .rego files stored in cvat/apps/*/rules/. Rego is a declarative language employed for defining OPA (Open Policy Agent) policies, and its syntax is detailed in the OPA documentation.

Note: Once you’ve made changes to the .rego files, you must rebuild and restart the Docker Compose for those changes to be applied. In this scenario, be sure to include the docker-compose.dev.yml compose configuration file when executing the Docker Compose command.

See:

• Global roles in CVAT.ai
• Organization roles in CVAT.ai
• Job Stage

Global roles in CVAT.ai

Note: Global roles can be adjusted only on self-hosted solution.

CVAT has implemented three Global roles, categorized as user Groups. These roles are:

Organization roles in CVAT.ai

Organization Roles are available only within the CVAT Organization.

Organization roles are assigned when users are invited to the Organization.

There are the following roles available in CVAT:

Job Stage

Job Stage can be assigned to any team member.

Stages are not roles.

Jobs can have an assigned user (with any role) and that Assignee will perform a Stage specific work which is to annotate, validate, or accept the job.

Job Stage can be:

Any Assignee can modify their assigned Stage specific functions via the annotation interface toolbar:

• Standard: switches interface to Annotation mode.
• Review: switches interface to the Validation mode.

5.2.7 - Track mode (advanced)

Basic operations in the mode were described in section track mode (basics).

Shapes that were created in the track mode, have extra navigation buttons.

• These buttons help to jump to the previous/next keyframe.

  

• The button helps to jump to the initial frame and to the last keyframe.

  

You can use the Split function to split one track into two tracks:

5.2.8 - 3D Object annotation (advanced)

As well as 2D-task objects, 3D-task objects support the ability to change appearance, attributes, properties and have an action menu. Read more in objects sidebar section.

Moving an object

If you hover the cursor over a cuboid and press Shift+N, the cuboid will be cut, so you can paste it in other place (double-click to paste the cuboid).

Copying

As well as in 2D task you can copy and paste objects by Ctrl+C and Ctrl+V, but unlike 2D tasks you have to place a copied object in a 3D space (double click to paste).

Image of the projection window

You can copy or save the projection-window image by left-clicking on it and selecting a “save image as” or “copy image”.

5.2.9 - Attribute annotation mode (advanced)

Basic operations in the mode were described in section attribute annotation mode (basics).

It is possible to handle lots of objects on the same frame in the mode.

It is more convenient to annotate objects of the same type. In this case you can apply the appropriate filter. For example, the following filter will hide all objects except person: label=="Person".

To navigate between objects (person in this case), use the following buttons switch between objects in the frame on the special panel:

or shortcuts:

• Tab — go to the next object
• Shift+Tab — go to the previous object.

In order to change the zoom level, go to settings (press F3) in the workspace tab and set the value Attribute annotation mode (AAM) zoom margin in px.

5.2.10 - Annotation with rectangles

To learn more about annotation using a rectangle, see the sections:

• Shape mode (basics)
• Track mode (basics)
• Shape mode (advanced)
• Track mode (advanced)

Rotation rectangle

To rotate the rectangle, pull on the rotation point. Rotation is done around the center of the rectangle. To rotate at a fixed angle (multiple of 15 degrees), hold shift. In the process of rotation, you can see the angle of rotation.

Annotation with rectangle by 4 points

It is an efficient method of bounding box annotation, proposed here. Before starting, you need to make sure that the drawing method by 4 points is selected.

Press Shape or Track for entering drawing mode. Click on four extreme points: the top, bottom, left- and right-most physical points on the object. Drawing will be automatically completed right after clicking the fourth point. Press Esc to cancel editing.

5.2.11 - Annotation with polygons

5.2.11.1 - Manual drawing

It is used for semantic / instance segmentation.

Before starting, you need to select Polygon on the controls sidebar and choose the correct Label.

• Click Shape to enter drawing mode. There are two ways to draw a polygon: either create points by clicking or by dragging the mouse on the screen while holding Shift.

Clicking points	Holding Shift+Dragging

• When Shift isn’t pressed, you can zoom in/out (when scrolling the mouse wheel) and move (when clicking the mouse wheel and moving the mouse), you can also delete the previous point by right-clicking on it.
• You can use the Selected opacity slider in the Objects sidebar to change the opacity of the polygon. You can read more in the Objects sidebar section.
• Press N again or click the Done button on the top panel for completing the shape.
• After creating the polygon, you can move the points or delete them by right-clicking and selecting Delete point or clicking with pressed Alt key in the context menu.

5.2.11.2 - Drawing using automatic borders

You can use auto borders when drawing a polygon. Using automatic borders allows you to automatically trace the outline of polygons existing in the annotation.

• To do this, go to settings -> workspace tab and enable Automatic Bordering or press Ctrl while drawing a polygon.

  

• Start drawing / editing a polygon.

• Points of other shapes will be highlighted, which means that the polygon can be attached to them.

• Define the part of the polygon path that you want to repeat.

  

• Click on the first point of the contour part.

  

• Then click on any point located on part of the path. The selected point will be highlighted in purple.

  

• Click on the last point and the outline to this point will be built automatically.

  

Besides, you can set a fixed number of points in the Number of points field, then drawing will be stopped automatically. To enable dragging you should right-click inside the polygon and choose Switch pinned property.

Below you can see results with opacity and black stroke:

If you need to annotate small objects, increase Image Quality to 95 in Create task dialog for your convenience.

5.2.11.3 - Edit polygon

To edit a polygon you have to click on it while holding Shift, it will open the polygon editor.

• In the editor you can create new points or delete part of a polygon by closing the line on another point.

• When Intelligent polygon cropping option is activated in the settings, CVAT considers two criteria to decide which part of a polygon should be cut off during automatic editing.

  • The first criteria is a number of cut points.
  • The second criteria is a length of a cut curve.

  If both criteria recommend to cut the same part, algorithm works automatically, and if not, a user has to make the decision. If you want to choose manually which part of a polygon should be cut off, disable Intelligent polygon cropping in the settings. In this case after closing the polygon, you can select the part of the polygon you want to leave.

  

• You can press Esc to cancel editing.

  

5.2.11.4 - Track mode with polygons

Polygons in the track mode allow you to mark moving objects more accurately other than using a rectangle (Tracking mode (basic); Tracking mode (advanced)).

1. To create a polygon in the track mode, click the Track button.

   

2. Create a polygon the same way as in the case of Annotation with polygons. Press N or click the Done button on the top panel to complete the polygon.

3. Pay attention to the fact that the created polygon has a starting point and a direction, these elements are important for annotation of the following frames.

4. After going a few frames forward press Shift+N, the old polygon will disappear and you can create a new polygon. The new starting point should match the starting point of the previously created polygon (in this example, the top of the left mirror). The direction must also match (in this example, clockwise). After creating the polygon, press N and the intermediate frames will be interpolated automatically.

   

5. If you need to change the starting point, right-click on the desired point and select Set starting point. To change the direction, right-click on the desired point and select switch orientation.

   

There is no need to redraw the polygon every time using Shift+N, instead you can simply move the points or edit a part of the polygon by pressing Shift+Click.

5.2.11.5 - Creating masks

Cutting holes in polygons

Currently, CVAT does not support cutting transparent holes in polygons. However, it is poissble to generate holes in exported instance and class masks. To do this, one needs to define a background class in the task and draw holes with it as additional shapes above the shapes needed to have holes:

The editor window:

Remember to use z-axis ordering for shapes by [-] and [+, =] keys.

Exported masks:

Notice that it is currently impossible to have a single instance number for internal shapes (they will be merged into the largest one and then covered by “holes”).

Creating masks

There are several formats in CVAT that can be used to export masks:

• Segmentation Mask (PASCAL VOC masks)
• CamVid
• MOTS
• ICDAR
• COCO (RLE-encoded instance masks, guide)
• Datumaro

An example of exported masks (in the Segmentation Mask format):

Important notices:

• Both boxes and polygons are converted into masks
• Grouped objects are considered as a single instance and exported as a single mask (label and attributes are taken from the largest object in the group)

Class colors

All the labels have associated colors, which are used in the generated masks. These colors can be changed in the task label properties:

Label colors are also displayed in the annotation window on the right panel, where you can show or hide specific labels (only the presented labels are displayed):

A background class can be:

• A default class, which is implicitly-added, of black color (RGB 0, 0, 0)
• background class with any color (has a priority, name is case-insensitive)
• Any class of black color (RGB 0, 0, 0)

To change background color in generated masks (default is black), change background class color to the desired one.

5.2.12 - Annotation with polylines

It is used for road markup annotation etc.

Before starting, you need to select the Polyline. You can set a fixed number of points in the Number of points field, then drawing will be stopped automatically.

Click Shape to enter drawing mode. There are two ways to draw a polyline — you either create points by clicking or by dragging a mouse on the screen while holding Shift. When Shift isn’t pressed, you can zoom in/out (when scrolling the mouse wheel) and move (when clicking the mouse wheel and moving the mouse), you can delete previous points by right-clicking on it. Press N again or click the Done button on the top panel to complete the shape. You can delete a point by clicking on it with pressed Ctrl or right-clicking on a point and selecting Delete point. Click with pressed Shift will open a polyline editor. There you can create new points(by clicking or dragging) or delete part of a polygon closing the red line on another point. Press Esc to cancel editing.

5.2.13 - Annotation with points

5.2.13.1 - Points in shape mode

It is used for face, landmarks annotation etc.

Before you start you need to select the Points. If necessary you can set a fixed number of points in the Number of points field, then drawing will be stopped automatically.

Click Shape to entering the drawing mode. Now you can start annotation of the necessary area. Points are automatically grouped — all points will be considered linked between each start and finish. Press N again or click the Done button on the top panel to finish marking the area. You can delete a point by clicking with pressed Ctrl or right-clicking on a point and selecting Delete point. Clicking with pressed Shift will open the points shape editor. There you can add new points into an existing shape. You can zoom in/out (when scrolling the mouse wheel) and move (when clicking the mouse wheel and moving the mouse) while drawing. You can drag an object after it has been drawn and change the position of individual points after finishing an object.

5.2.13.2 - Linear interpolation with one point

You can use linear interpolation for points to annotate a moving object:

1. Before you start, select the Points.

2. Linear interpolation works only with one point, so you need to set Number of points to 1.

3. After that select the Track.

   

4. Click Track to enter the drawing mode left-click to create a point and after that shape will be automatically completed.

   

5. Move forward a few frames and move the point to the desired position, this way you will create a keyframe and intermediate frames will be drawn automatically. You can work with this object as with an interpolated track: you can hide it using the Outside, move around keyframes, etc.

   

6. This way you’ll get linear interpolation using the Points.

   

5.2.14 - Annotation with ellipses

It is used for road sign annotation etc.

First of all you need to select the ellipse on the controls sidebar.

Choose a Label and click Shape or Track to start drawing. An ellipse can be created the same way as a rectangle, you need to specify two opposite points, and the ellipse will be inscribed in an imaginary rectangle. Press N or click the Done button on the top panel to complete the shape.

You can rotate ellipses using a rotation point in the same way as rectangles.

Annotation with ellipses video tutorial

5.2.15 - Annotation with cuboids

It is used to annotate 3 dimensional objects such as cars, boxes, etc… Currently the feature supports one point perspective and has the constraint where the vertical edges are exactly parallel to the sides.

5.2.15.1 - Creating the cuboid

Before you start, you have to make sure that Cuboid is selected and choose a drawing method ”from rectangle” or “by 4 points”.

Drawing cuboid by 4 points

Choose a drawing method “by 4 points” and click Shape to enter the drawing mode. There are many ways to draw a cuboid. You can draw the cuboid by placing 4 points, after that the drawing will be completed automatically. The first 3 points determine the plane of the cuboid while the last point determines the depth of that plane. For the first 3 points, it is recommended to only draw the 2 closest side faces, as well as the top and bottom face.

A few examples:

Drawing cuboid from rectangle

Choose a drawing method “from rectangle” and click Shape to enter the drawing mode. When you draw using the rectangle method, you must select the frontal plane of the object using the bounding box. The depth and perspective of the resulting cuboid can be edited.

Example:

5.2.15.2 - Editing the cuboid

The cuboid can be edited in multiple ways: by dragging points, by dragging certain faces or by dragging planes. First notice that there is a face that is painted with gray lines only, let us call it the front face.

You can move the cuboid by simply dragging the shape behind the front face. The cuboid can be extended by dragging on the point in the middle of the edges. The cuboid can also be extended up and down by dragging the point at the vertices.

To draw with perspective effects it should be assumed that the front face is the closest to the camera. To begin simply drag the points on the vertices that are not on the gray/front face while holding Shift. The cuboid can then be edited as usual.

If you wish to reset perspective effects, you may right click on the cuboid, and select Reset perspective to return to a regular cuboid.

The location of the gray face can be swapped with the adjacent visible side face. You can do it by right clicking on the cuboid and selecting Switch perspective orientation. Note that this will also reset the perspective effects.

Certain faces of the cuboid can also be edited, these faces are: the left, right and dorsal faces, relative to the gray face. Simply drag the faces to move them independently from the rest of the cuboid.

You can also use cuboids in track mode, similar to rectangles in track mode (basics and advanced) or Track mode with polygons

5.2.16 - Annotation with skeletons

In this guide, we delve into the efficient process of annotating complex structures through the implementation of Skeleton annotations.

Skeletons serve as annotation templates for annotating complex objects with a consistent structure, such as human pose estimation or facial landmarks.

A Skeleton is composed of numerous points (also referred to as elements), which may be connected by edges. Each point functions as an individual object, possessing unique attributes and properties like color, occlusion, and visibility.

Skeletons can be exported in two formats: CVAT for image and COCO Keypoints.

Note: that skeletons’ labels cannot be imported in a label-less project by importing a dataset. You need to define the labels manually before the import.

See:

• Adding Skeleton manually
  • Skeleton Configurator
  • Configuring Skeleton points
  • Adding Skeleton labels manually
• Adding Skeleton labels from the model
• Annotation with Skeletons
• Automatic annotation with Skeletons
• Editing skeletons on the canvas
• Editing skeletons on the sidebar

Adding Skeleton manually

To start annotating using skeletons, you need to set up a Skeleton task in Configurator:

To open Configurator, when creating a task, click on the Setup skeleton button if you want to set up the skeleton manually, or From model if you want to add skeleton labels from a model.

Skeleton Configurator

The skeleton Configurator is a tool to build skeletons for annotation. It has the following fields:

Configuring Skeleton points

You can name labels, set attributes, and change the color of each point of the skeleton.

To do this, right-click on the skeleton point and select Configure:

In the opened menu, you can change the point setting. It is similar to adding labels and attributes of the regular task:

A Skeleton point can only exist within its parent Skeleton.

Note that you cannot change the skeleton configuration for an existing task/project.

You can copy/insert skeleton configuration from the Raw tab of the label configurator.

Adding Skeleton labels manually

To create the Skeleton task, do the following:

1. Open Configurator.
2. (Optional) Upload background image.
3. In the Label name field, enter the name of the label.
4. (Optional) Add attribute
   Note: you can add attributes exclusively to each point, for more information, see Configuring Skeleton points
5. Use Add point to add points to the Drawing area.
6. Use Add edge to add edges between points.
7. Upload files.
8. Click:
   • Submit & Open to create and open the task.
   • Submit & Continue to submit the configuration and start creating a new task.

Adding Skeleton labels from the model

To add points from the model, and annotate do the following:

1. Open Basic configurator.

2. On the Constructor tab, click From model.

3. From the Select a model to pick labels select the Human pose estimation model or others if available.

4. Click on the model’s labels, you want to use.
   Selected labels will become gray.

   

5. (Optional) If you want to adjust labels, within the label, click the Update attributes icon.
   The Skeleton configurator will open, where you can configure the skeleton.
   Note: Labels cannot be adjusted after the task/project is created.

6. Click Done. The labels, that you selected, will appear in the labels window.

7. Upload data.

8. Click:

   • Submit & Open to create and open the task.
   • Submit & Continue to submit the configuration and start creating a new task.

Annotation with Skeletons

To annotate with Skeleton, do the following

1. Open job.

2. On the tools panel select Draw new skeleton.

3. Select Track or Shape to annotate. without tracking.

   

4. Draw a skeleton on the image.

Automatic annotation with Skeletons

To automatically annotate with Skeleton, do the following

1. Open the job and on the tools panel select AI Tools > Detectors

2. From the drop-down list select the model. You will see a list of points to match and the name of the skeleton on the top of the list.

   

3. (Optional) By clicking on the Bin icon, you can remove any mapped item:

   • A skeleton together with all points.
   • Certain points from two mapped skeletons.

4. Click Annotate.

Editing skeletons on the canvas

A drawn skeleton is encompassed within a bounding box, it allows you to manipulate the skeleton as a regular bounding box, enabling actions such as dragging, resizing, or rotating:

Upon repositioning a point, the bounding box adjusts automatically, without affecting other points:

Additionally, Shortcuts are applicable to both the skeleton as a whole and its elements:

• To use a shortcut to the entire skeleton, hover over the bounding box and push the shortcut keyboard key. This action is applicable for shortcuts like the lock, occluded, pinned, keyframe, and outside for skeleton tracks.
• To use a shortcut to a specific skeleton point, hover over the point and push the shortcut keyboard key. The same list of shortcuts is available, with the addition of outside, which is also applicable to individual skeleton shape elements.

Editing skeletons on the sidebar

In CVAT, the sidebar offers an alternative method for setting up skeleton properties and attributes.

This approach is similar to that used for other object types supported by CVAT, but with a few specific alterations:

An additional collapsible section is provided for users to view a comprehensive list of skeleton parts.

Skeleton points can have properties like Outside, Occluded, and Hidden.

Both Outside and Hidden make a skeleton point invisible.

• Outside property is part of annotations. Use it when part of the object is out of frame borders.

• Hidden makes a point hidden only for the annotator’s convenience, this property will not be saved between different sessions.

• Occluded keeps the point visible on the frame and usually means that the point is still on a frame, just hidden behind another object.

5.2.17 - Annotation with brush tool

With a brush tool, you can create masks for disjoint objects, that have multiple parts, such as a house hiding behind trees, a car behind a pedestrian, or a pillar behind a traffic sign. The brush tool has several modes, for example: erase pixels, change brush shapes, and polygon-to-mask mode.

Use brush tool for Semantic (Panoptic) and Instance Image Segmentation tasks.
For more information about segmentation masks in CVAT, see Creating masks.

See:

• Brush tool menu
• Annotation with brush
• Annotation with polygon-to-mask
• Remove underlying pixels
• AI Tools
• Import and export

Brush tool menu

The brush tool menu appears on the top of the screen after you click Shape:

It has the following elements:

Element	Description
	Save mask saves the created mask. The saved mask will appear on the object sidebar
	Save mask and continue adds a new mask to the object sidebar and allows you to draw a new one immediately.
	Brush adds new mask/ new regions to the previously added mask).
	Eraser removes part of the mask.
	Polygon selection tool. Selection will become a mask.
	Remove polygon selection subtracts part of the polygon selection.
	Brush size in pixels. Note: Visible only when Brush or Eraser are selected.
	Brush shape with two options: circle and square. Note: Visible only when Brush or Eraser are selected.
	Remove underlying pixels. When you are drawing or editing a mask with this tool, pixels on other masks that are located at the same positions as the pixels of the current mask are deleted.
	Hide mask. When drawing or editing a mask, you can enable this feature to temporarily hide the mask, allowing you to see the objects underneath more clearly.
	Label that will be assigned to the newly created mask
	Move. Click and hold to move the menu bar to the other place on the screen

Annotation with brush

To annotate with brush, do the following:

1. From the controls sidebar, select Brush .

2. In the Draw new mask menu, select label for your mask, and click Shape.
   The Brush tool will be selected by default.

   

3. With the brush, draw a mask on the object you want to label.
   To erase selection, use Eraser

   

4. After you applied the mask, on the top menu bar click Save mask
   to finish the process (or N on the keyboard).

5. Added object will appear on the objects sidebar.

To add the next object, repeat steps 1 to 5. All added objects will be visible on the image and the objects sidebar.

To save the job with all added objects, on the top menu, click Save .

Annotation with polygon-to-mask

To annotate with polygon-to-mask, do the following:

1. From the controls sidebar, select Brush .

2. In the Draw new mask menu, select label for your mask, and click Shape.

   

3. In the brush tool menu, select Polygon .

4. With the Polygon tool, draw a mask for the object you want to label.
   To correct selection, use Remove polygon selection .

5. Use Save mask (or N on the keyboard)
   to switch between add/remove polygon tools:

   

6. After you added the polygon selection, on the top menu bar click Save mask
   to finish the process (or N on the keyboard).

7. Click Save mask again (or N on the keyboard).
   The added object will appear on the objects sidebar.

To add the next object, repeat steps 1 to 5.

All added objects will be visible on the image and the objects sidebar.

To save the job with all added objects, on the top menu, click Save .

Remove underlying pixels

Use Remove underlying pixels tool when you want to add a mask and simultaneously delete the pixels of
other masks that are located at the same positions. It is a highly useful feature to avoid meticulous drawing edges twice between two different objects.

AI Tools

You can convert AI tool masks to polygons. To do this, use the following AI tool menu:

1. Go to the Detectors tab.
2. Switch toggle Masks to polygons to the right.
3. Add source and destination labels from the drop-down lists.
4. Click Annotate.

Import and export

For export, see Export dataset

Import follows the general import dataset procedure, with the additional option of converting masks to polygons.

Note: This option is available for formats that work with masks only.

To use it, when uploading the dataset, switch the Convert masks to polygon toggle to the right:

5.2.18 - Annotation with tags

It is used to annotate frames, tags are not displayed in the workspace. Before you start, open the drop-down list in the top panel and select Tag annotation.

The objects sidebar will be replaced with a special panel for working with tags. Here you can select a label for a tag and add it by clicking on the Plus button. You can also customize hotkeys for each label.

If you need to use only one label for one frame, then enable the Automatically go to the next frame checkbox, then after you add the tag the frame will automatically switch to the next.

Tags will be shown in the top left corner of the canvas. You can show/hide them in the settings.

5.2.19 - Models

To deploy the models, you will need to install the necessary components using Semi-automatic and Automatic Annotation guide. To learn how to deploy the model, read Serverless tutorial.

The Models page contains a list of deep learning (DL) models deployed for semi-automatic and automatic annotation. To open the Models page, click the Models button on the navigation bar. The list of models is presented in the form of a table. The parameters indicated for each model are the following:

• Framework the model is based on
• model Name
• model Type:
  • detector - used for automatic annotation (available in detectors and automatic annotation)
  • interactor - used for semi-automatic shape annotation (available in interactors)
  • tracker - used for semi-automatic track annotation (available in trackers)
  • reid - used to combine individual objects into a track (available in automatic annotation)
• Description - brief description of the model
• Labels - list of the supported labels (only for the models of the detectors type)

5.2.20 - CVAT Analytics and QA in Cloud

5.2.20.1 - Automated QA, Review & Honeypots

In CVAT, it’s possible to evaluate the quality of annotation through the creation of a validation subset of images. To estimate the task quality, CVAT compares all other jobs in the task against the established Ground truth job, and calculates annotation quality based on this comparison.

Note that quality estimation only supports 2d tasks. It supports all the annotation types except 2d cuboids.

Note that quality estimation is currently available for tasks and jobs. Quality estimation in projects is not supported.

CVAT has the following features for automated quality control of annotations:

• Validation set configuration for a task
• Job validation on job finish (“Immediate feedback”)
• Review mode for problems found
• Quality analytics

Basics

There are several approaches to quality estimation used in the industry. In CVAT, we can use a method known as Ground Truth or Honeypots. The method assumes there are Ground Truth annotations for images in the dataset. This method is statistical, which means that we can use only a small portion of the whole dataset to estimate quality on the full dataset, so we don’t need to annotate the whole dataset twice. Here we assume that the images in the dataset are similar (represent the same task).

We will call the validation portion of the whole dataset (or a task in CVAT) a validation set. In practice, it is typically expected that annotations in the validation set are carefully validated and curated. It means that they are more expensive - creating them might require expert annotators or just several iterations of annotation and validation. It means that it’s desirable to keep the validation set small enough. At the same time, it must be representative enough to provide reliable estimations. To achieve this, it’s advised that the validation set images are sampled randomly and independently from the full dataset. That is, for the quality assurance to function correctly, the validation set must have some portion of the task frames, and the frames must be chosen randomly.

Depending on the dataset size, data variance, and task complexity, 5-15% of the data is typically good enough for quality estimation, while keeping extra annotation overhead for the Ground Truth acceptable.

For example, in a typical task with 2000 frames, selecting just 5%, which is 100 extra frames to annotate, is enough to estimate the annotation quality. If the task contains only 30 frames, it’s advisable to select 8-10 frames, which is about 30%. It is more than 15%, but in the case of smaller datasets, we need more samples to estimate quality reliably, as data variance is higher.

Ground truth jobs

A Ground Truth job (GT job) is a way to represent the validation set in a CVAT task. This job is similar to regular annotation jobs - you can edit the annotations manually, use auto-annotation features, and import annotations in this job. There can be no more than 1 Ground Truth job in a task.

To enable quality estimation in a task, you need to create a Ground truth job in the task, annotate it, switch the job stage to acceptance, and set the job state to completed. Once the Ground Truth job is configured, CVAT will start using this job for quality estimation.

Read more about Ground Truth management here.

Configuring quality estimation

Quality estimation is configured on the Task level.

• In a new task
• In an existing task

A Ground truth job is considered configured if it is at the acceptance stage and in the completed state.

A configured Ground Truth job is required for all quality computations in CVAT.

Validation modes

Currently, there are 2 validation modes available for tasks: Ground Truth and Honeypots. These names are often used interchangeably, but in CVAT they have some differences. Both modes rely on the use of Ground Truth annotations in a task, stored in a Ground Truth job, where they can be managed.

Ground Truth

In this mode some of the task frames are selected into the validation set, represented as a separate Ground Truth job. The regular annotation jobs in the task are not affected in any way.

Ground Truth jobs can be created at the task creation automatically or manually at any moment later. They can also be removed manually at any moment. This validation mode is available for any tasks and annotations.

This is a flexible mode that can be enabled or disabled at any moment without any disruptions to the annotation process.

Frame selection

This validation mode can use several frame selection methods.

Random

This is a simple method that selects frames into the validation set randomly, representing the basic approach, described above.

Parameters:

• frame count - the number or percent of the task frames to be used for validation. Can be specified as an absolute number in the Frame count field or a percent in the Quantity field. If there are both fields on the page, they are linked, which means changing one of them will adjust the other one automatically.
• random seed - a number to be used to initialize the random number generator. Can be useful if you want to create a reproducible sequence of frames.

Random per job

This method selects frames into the validation set randomly from each annotation job in the task.

It solves one of the issues with the simple Random method that some of the jobs can get no validation frames, which makes it impossible to estimate quality in such jobs. Note that using this method can result in increased total size of the validation set.

Parameters:

• frame count per job - the percent of the job frames to be used for validation. This method uses segment size of the task to select the same number of validation frames in each job, if possible. Can be specified as an absolute number in the Frame count field or a percent in the Quantity per job field. If there are both fields on the page, they are linked, which means changing one of them will adjust the other one automatically.
• random seed - a number to be used to initialize the random number generator. Can be useful if you want to create a reproducible sequence of frames.

Honeypots

In this mode some random frames of the task are selected into the validation set. Then, validation frames are randomly mixed into regular annotation jobs. This mode can also be called “Ground Truth pool”, reflecting the way validation frames are used. This mode can only be used at task creation and cannot be changed later.

The mode has some limitations on the compatible tasks:

• It’s not possible to use it for an already existing task, the task has to be recreated.
• This mode assumes random frame ordering, so it is only available for image annotation tasks and not for ordered sequences like videos.
• Tracks are not supported in such tasks.

The validation set can be managed after the task is created - annotations can be edited, frames can be excluded and restored, and honeypot frames in the regular jobs can be changed. However, it’s not possible to select new validation frames after the task is created. The Ground truth job created for this validation mode cannot be deleted.

Parameters:

• frame count per job (%) - the percent of job frames (segment size) to be added into each annotation job from the validation set. Can be specified in the Overhead per job field.
• total frame count (%) - the percent of the task frames to be included into the validation set. This value must result in at least frame count per job * segment size frames. Can be specified in the Total honeypots field.

Mode summary

Here is a brief comparison of the validation modes:

Choosing the right mode

Here are some examples on how to choose between these options. The general advice is to use Ground Truth for better flexibility, but keep in mind that it can require more resources for validation set annotation. Honeypots, on the other hand, can be beneficial if you want to minimize the number of validation images required, but the downside here is that there are some limitations on where this mode can be used.

Example: a video annotation with tracks. In this case there is only 1 option - the Ground Truth mode, so just use it.

Example: an image dataset annotation, image order is not important. Here you can use both options. You can choose Ground Truth for better flexibility in validation. This way, you will have the full control of validation frames in the task, annotation options won’t be limited, and the regular jobs will not be affected in any way. However, if you have a limited budget for the validation (for instance, you have only a small number of validation frames) or you want to allow more scalability (with this approach the number of validation frames doesn’t depend on the number of regular annotation jobs), it makes sense to consider using Honeypots instead.

Quality management

If a task has a validation configured, there are several options to manage validation set images. With any of the validation modes, there will be a special Ground Truth (GT) job in the task.

Validation set management

Validation frames can be managed on the task Quality Management page. Here it’s possible to check the number of validation frames, current validation mode and review the frame details. For each frame you can see the number of uses in the task. When in the Ground Truth mode, this number will be 1 for all frames. With Honeypots, these numbers can be 0, 1 or more.

Frame changes

In both validation modes it’s possible to exclude some of the validation frames from being used for validation. This can be useful if you find that some of the validation frames are “bad”, extra, or if they have incorrect annotations, which you don’t want to fix. Once a frame is marked “excluded”, it will not be used for validation. There is also an option to restore a previously excluded frame if you decide so.

There is an option to exclude or restore frames in bulk mode. To use it, select the frames needed using checkboxes, and click one of the buttons next to the table header.

Ground Truth job management

In the Ground Truth validation mode, there will be an option to remove the Ground Truth job from the task. It can be useful if you want to change validation set frames completely, add more frames, or remove some of the frames for any reason. This is available in the job Actions menu.

In the Honeypots mode, it’s not possible to add or remove the GT job, so it’s not possible to add more validation frames.

Create

A Ground Truth job can be added manually in a task without a selected validation mode or in a task with the Ground Truth validation mode, after the existing Ground Truth job is deleted manually.

Delete

To delete the Ground Truth job, do the following:

1. Open the task and find the Ground Truth job in the jobs list.
2. Click on three dots to open the menu.
3. From the menu, select Delete.

Note: The Ground truth job in the “Honeypots” task validation mode cannot be deleted.

Import annotations

If you want to import annotations into the Ground truth job, do the following:

1. Open the task and find the Ground truth job in the jobs list.
2. Click on three dots to open the menu.
3. From the menu, select Import annotations.
4. Select import format and select file.
5. Click OK.

Note that if there are imported annotations for the frames that exist in the task, but are not included in the Ground truth job, they will be ignored. This way, you don’t need to worry about “cleaning up” your Ground truth annotations for the whole dataset before importing them. Importing annotations for the frames that are not known in the task still raises errors.

Export annotations

To export annotations from the Ground Truth job, do the following:

1. Open the task and find a job in the jobs list.
2. Click on three dots to open the menu.
3. From the menu, select Export annotations.

Annotation management

Annotations for validation frames can be displayed and edited in a special Ground Truth job in the task. You can edit the annotations manually, use auto-annotation features, import and export annotations in this job.

In the Ground Truth task validation mode, annotations of the ground Truth job do not affect other jobs in any way. The Ground Truth job is just a separate job, which can only be changed directly. Annotations from Ground truth jobs are not included in the dataset export, they also cannot be imported during task annotations import or with automatic annotation for the task.

In the Honeypots task validation mode, the annotations of the GT job also do not affect other jobs in any way. However, import and export of task annotations works differently. When importing task annotations, annotations for validation frames will be copied both into GT job frames and into corresponding honeypot frames in annotation jobs. When exporting task annotations, honeypot frames in annotation jobs will be ignored, and validation frames in the resulting dataset will get annotations from the GT job.

Note that it means that exporting from a task with honeypots and importing the results back will result in changed annotations on the honeypot frames. If you want to backup annotations, use a task backup or export job annotations instead.

Import and export of Ground Truth job annotations works the same way in both modes.

Ground Truth jobs are included in task backups, so can be saved and restored this way.

Import, Export, and Delete options are available from the Ground Truth job Actions menu. Read more.

Annotation quality settings

If you need to tweak some aspects of comparisons, you can do this from the Annotation Quality Settings menu.

You can configure what overlap should be considered low or how annotations must be compared.

The updated settings will take effect on the next quality update.

To open Annotation Quality Settings, find Quality report and on the right side of it, click on three dots.

The following window will open. Hover over the ? marks to understand what each field represents.

Annotation quality settings have the following parameters: