Annotation

• 1: Editor
• 1.1: Menu and Navigation Bar
• 1.2: Workspace
• 1.3: Objects sidebar
• 1.4: Controls sidebar
• 1.5: Settings
• 1.6: 3D task workspace
• 2: Tools & Shapes
• 2.1: Shape mode (advanced)
• 2.2: Single Shape
• 2.3: Track mode (advanced)
• 2.4: 3D Object annotation (advanced)
• 2.5: Attribute annotation mode (advanced)
• 2.6: Annotation with rectangles
• 2.7: Annotation with polygons
• 2.7.1: Manual drawing
• 2.7.2: Drawing using automatic borders
• 2.7.3: Edit polygon
• 2.7.4: Track mode with polygons
• 2.7.5: Creating masks
• 2.8: Annotation with polylines
• 2.9: Annotation with points
• 2.9.1: Points in shape mode
• 2.9.2: Linear interpolation with one point
• 2.10: Annotation with ellipses
• 2.11: Annotation with cuboids
• 2.11.1: Creating the cuboid
• 2.11.2: Editing the cuboid
• 2.12: Annotation with skeletons
• 2.13: Annotation with brush tool
• 2.14: Annotation with tags
• 2.15: AI Tools
• 2.16: Standard 3D mode (basics)
• 2.17: Types of shapes
• 2.18: Backup Task and Project
• 2.19: Shape mode (basics)
• 2.20: Frame deleting
• 2.21: Join and slice tools
• 2.22: Track mode (basics)
• 2.23: 3D Object annotation
• 2.24: Attribute annotation mode (basics)
• 2.25: Filter
• 2.26: Contextual images
• 2.27: Shape grouping
• 2.28: Dataset Manifest
• 2.29: Data preparation on the fly
• 2.30: Shapes converter
• 3: Automated Annotation
• 3.1: Overview
• 3.2: Segment Anything 2 Tracker

1 - Editor

This section of the documentation describes the annotation interface and all available options that you can use to annotate image data accurately and quickly.

The interface includes the following areas:

1.1 - Menu and Navigation Bar

The navigation panel and drop-down Menu, allow you to switch between frames, change the annotation mode, save your work, and more.

See:

• Menu
• Navigation bar
  • Save, Undo, Done
  • Navigation controls
• Job information and Annotation Mode switcher

Menu

Use the Menu options to upload and download annotations, change the status of the job, and access other features listed in the table below:

Panel Item	Description
Upload annotations	Upload annotations into a task.
Export as a dataset	Download a dataset in one of the supported formats.
Remove annotations	Delete all annotations for the current job. Use Select range to remove annotations for a specific range of frames. Enable the Delete only keyframe for tracks checkbox to delete only keyframes from the tracks within the selected range.
Run actions	Run annotation actions on the annotated dataset. Annotations action is a feature that allows you to modify a bulk of annotations on many frames. It supports only shape objects.
Open the task	Opens a page with details about the task.
Change job state	Changes the state of the job: New: The job is newly created and has not been started yet. It is waiting for annotation work to begin. In Progress: The job is currently being worked on. Annotations are being added or edited. Rejected: The job has been reviewed and deemed unsatisfactory or incorrect. It requires revisions and further work. Completed: The job has been finished, reviewed, and approved. No further changes are necessary.
Finish the job	Saves annotations and sets job state to Completed.

Navigation bar

Use the navigation bar to save annotation results, switch between frames, and access other features listed in the tables below.

Save, Undo, Done

Use the following buttons, to save your work, undo changes, and move tasks to done.

Function	Description
Save work	Saves annotations for the current job. The button indicates the saving process.
Undo/Redo	Use buttons to undo actions or redo them.
Done	Used to complete the creation of the object. This button appears only when the object is being created.
Block	Used to pause automatic line creation when drawing a polygon with OpenCV Intelligent scissors. Also used to postpone server requests when creating an object using AI Tools.

Navigation controls

Overview of how to navigate through frames within the interface, with detailed descriptions provided in the table below.

Function	Description
Go to the first/last frame	Navigate to the first or the last frame of the sequence.
Go back with a step/Go next with a step	Move to the previous or next frame by a predefined step. Shortcuts: C — previous frame. V — next frame. Default step size is 10 frames. To modify this, navigate to Nickname > Settings > Player Step.
Go back/Go next	Navigate to the neighboring frames. Shortcuts: D — previous frame. F — next frame. Go back/Go next buttons are customizable: To customize, right-click on the button and select one of three options (left to right): The default setting moves to the next or previous frame (step size is 1 frame). Move to the next or previous frame that contains objects (e.g., filtered). For more information, refer to Filters. Move to the next or previous frame without annotations. Use this option to quickly locate missed frames. .
Play/Pause	Switch between playing and pausing the sequence of frames or set of images. Shortcut: Space. To adjust the playback speed, go to Nickname > Settings > Player Speed.
Go to the specific frame	Enter the number of the frame you want to go to and press Enter.
Search frame by filename	Click to open the search pop-up. Type a frame filename to search for it within the job. Select the filename and press Enter to navigate to the selected frame.
Copy frame name	Click to copy frame name to the clipboard.
Copy frame link	Click to copy link to the frame.
Delete frame	Click to delete or restore current frame.

Job information and Annotation Mode switcher

This section outlines various functionalities, including how to switch to the fullscreen player mode, access job information, and use the Workspace Switcher to toggle between different annotation and QA modes.

Function	Description
Fullscreen	The fullscreen player mode. The keyboard shortcut is F11.
Info	Open the job info. Overview: Assignee - the individual to whom the job is assigned. Reviewer– the user tasked with conducting the review. For more information, see Manual QA Start frame - the number of the first frame in this job. Stop frame - the number of the last frame in this job. Frames - the total number of frames in the job. Annotations Statistics table displays the number of created shapes, categorized by labels (e.g., vehicle, person) and the type of annotation (shape, track), as well as the count of manual and interpolated frames.
Filters	Switches on Filters.
Workplace Switcher	The drop-down list to switch between different annotation modes: Overview: Standard – default mode. Attribute – annotation with Attributes Single Shape – Single shape annotation mode. Tag annotation- annotation with Tags Review – Manual QA mode.

1.2 - Workspace

In CVAT, the workspace serves as a work area where annotators interact with images, videos, and the various tools available to create high-quality annotations.

See:

• Image settings in CVAT
  • Adding grid overlay to image in CVAT
  • Changing color settings of image in CVAT
• Adding layers and Z-axis slider
• Interacting with Objects

Image settings in CVAT

The Image settings panel serves as a versatile tool for fine-tuning the visual aspects of your image. Whether you need to brighten the image, increase contrast, or make other adjustments, this panel is your go-to.

Additionally, the panel allows you to overlay a grid on the image for more precise annotation.

By default, the Image settings panel is not visible. To access it, click on the Arrow Up () icon located at the bottom of the workspace.

Adding grid overlay to image in CVAT

To add the grid to the image, do the following:

1. Open the Image Settings panel.
2. Locate and check the box that allows you to overlay a grid on the image.
3. Specify the grid cell size in square millimeters by entering the desired number in the Size field.
4. From the Color drop-down list, select the color of the grid.
5. Use the Opacity slider to change the transparency of the grid overlay.

Changing color settings of image in CVAT

To change the color setting of the image is CVAT, do the following:

1. Open the Image Settings panel.
2. Use the slider to change the color quality.

There are four color quality settings in CVAT:

Brightness increases and decreases the overall lightness of the image:

Contrast is the range of brightness, from lightest to darkest, in an image.

Saturation describes the intensity of the color.

Gamma correction can be used to control the overall brightness of an image

To reset the setting to default values, click Reset color settings

Adding layers and Z-axis slider

Z-axis Slider enables you to add annotation layers while hiding the layers positioned beyond.

You can also move between layers by moving the slider to the layer you need.

The slider becomes active when multiple Z-layers are present within a frame. Click + on the slider to add a new layer; upon pressing it, a new layer is automatically created and activated.

You can also relocate objects between layers using the + and - keys.

Interacting with Objects

The workspace is also equipped with the following features:

• Right-clicking an object opens the Object Card. This interface contains essential controls for modifying the object’s label and attributes, as well as providing access to an action menu.

  

• Right-clicking on a polygon point will open a menu, from which you can Delete point or Set start point.

  

1.3 - Objects sidebar

In the objects sidebar, you can see the list of available objects on the current frame. The following figure is an example of how the list might look like:

Shape mode	Track mode

Objects properties

Filter input box

The way how to use filters is described in the advanced guide here.

List of objects

• Switch lock property for all - switches lock property of all objects in the frame.
• Switch hidden property for all - switches hide the property of all objects in the frame.
• Expand/collapse all - collapses/expands the details field of all objects in the frame.
• Sorting - sort the list of objects: updated time, ID - accent, ID - descent

Objects on the sidebar

The type of shape can be changed by selecting the Label property. For instance, it can look like shown in the figure below:

Object action menu

The action menu calls up the button:

The action menu contains:

• Create object URL - puts a link to an object on the clipboard. After you open the link, this object will be filtered.

• Make a copy - copies an object. The keyboard shortcut is Ctrl + C > Ctrl + V.

• Propagate function copies the form to multiple frames and displays a dialog box where you can specify the number of copies or the frame to which you want to copy the object. The keyboard shortcut is Ctrl + B. On how to propagate only filtered shapes, see Shapes converter
  There are two options available:

  • Propagate forward () creates a copy of the object on N subsequent frames at the same position.
  • Propagate backward () creates a copy of the object on N previous frames at the same position.

  

• To background - moves the object to the background. The keyboard shortcut - or _

• To foreground - moves the object to the foreground. The keyboard shortcut + or =

• Change instance color- choosing a color using the color picker (available only in instance mode).

  

• Remove - removes the object. The keyboard shortcut Del, Shift+Del.

A shape can be locked to prevent its modification or moving by an accident. Shortcut to lock an object: L.

A shape can be Occluded. Shortcut: Q. Such shapes have dashed boundaries.

You can change the way an object is displayed on a frame (show or hide).

Switch pinned property - when enabled, a shape cannot be moved by dragging or dropping.

Tracker switcher - enable/disable tracking for the object.

By clicking on the Details button you can collapse or expand the field with all the attributes of the object.

Labels

In this tab, you can lock or hide objects of a certain label. To change the color for a specific label, you need to go to the task page and select the color by clicking the edit button, this way you will change the label color for all jobs in the task.

Fast label change

You can change the label of an object using hotkeys. In order to do it, you need to assign a number (from 0 to 9) to labels. By default numbers 1,2…0 are assigned to the first ten labels. To assign a number, click on the button placed at the right of a label name on the sidebar.

After that, you will be able to assign a corresponding label to an object by hovering your mouse cursor over it and pressing Ctrl + Num(0..9).

In case you do not point the cursor to the object, pressing Ctrl + Num(0..9) will set a chosen label as default, so that the next object you create (use the N key) will automatically have this label assigned.

Appearance

Color By options

Change the color scheme of the annotation:

• Instance — every shape has a random color

  

• Group — every group of shape has its own random color, ungrouped shapes are white

  

• Label — every label (e.g. car, person) has its own random color

  

  You can change any random color pointing to a needed box on a frame or on an object sidebar.

Fill Opacity slider

Change the opacity of every shape in the annotation.

Selected Fill Opacity slider

Change the opacity of the selected object’s fill. It is possible to change the opacity while drawing an object in the case of rectangles, polygons, and cuboids.

Outlined borders checkbox

You can change a special shape border color by clicking on the Eyedropper icon.

Show bitmap checkbox

If enabled all shapes are displayed in white and the background is black.

Show projections checkbox

Enables/disables the display of auxiliary perspective lines. Only relevant for cuboids

Hide objects sidebar

Hide - the button hides the object’s sidebar.

1.4 - Controls sidebar

Navigation

Navigation block - contains tools for moving and rotating images.

Icon	Description
	Cursor (Esc)- a basic annotation editing tool.
	Move the image- a tool for moving around the image without the possibility of editing.
	Rotate- two buttons to rotate the current frame a clockwise (Ctrl+R) and anticlockwise (Ctrl+Shift+R). You can enable Rotate all images in the settings to rotate all the images in the job

Zoom

Zoom block - contains tools for image zoom.

Icon	Description
	Fit image- fits image into the workspace size. Shortcut - double click on an image
	Select a region of interest- zooms in on a selected region. You can use this tool to quickly zoom in on a specific part of the frame.

Shapes

Shapes block - contains all the tools for creating shapes.

Icon	Description	Links to section
	AI Tools	AI Tools
	OpenCV	OpenCV
	Rectangle	Shape mode; Track mode; Drawing by 4 points
	Polygon	Annotation with polygons; Track mode with polygons
	Polyline	Annotation with polylines
	Points	Annotation with points
	Ellipses	Annotation with ellipses
	Cuboid	Annotation with cuboids
	Brushing tools	Annotation with brushing
	Tag	Annotation with tags
	Open an issue	Review (available only in review mode)

Edit

Edit block - contains tools for editing tracks and shapes.

Icon	Description	Links to section
	Merge Shapes(M) - starts/stops the merging shapes mode.	Track mode (basics)
	Group Shapes (G) - starts/stops the grouping shapes mode.	Shape grouping
	Split - splits a track.	Track mode (advanced)
	Joins multiple labels into one	Joining mask tool
	Slices one label into several.	Slice mask/polygon

Move image

Switching between user interface modes.

1. Use arrows below to move to the next/previous frame. Use the scroll bar slider to scroll through frames. Almost every button has a shortcut. To get a hint about a shortcut, just move your mouse pointer over an UI element.

2. To navigate the image, use the button on the controls sidebar. Another way an image can be moved/shifted is by holding the left mouse button inside an area without annotated objects. If the Mouse Wheel is pressed, then all annotated objects are ignored. Otherwise the a highlighted bounding box will be moved instead of the image itself.

   

3. You can use the button on the sidebar controls to zoom on a region of interest. Use the button Fit the image to fit the image in the workspace. You can also use the mouse wheel to scale the image (the image will be zoomed relatively to your current cursor position).

   

1.5 - Settings

To open the Settings, open the user menu in the header and select Settings. Or you can use the default F2 shortcut.

The Settings section has three tabs:

• Player
• Workspace
• Shortcuts

Player

In the Player tab, you can:

• Control step of C and V shortcuts.
• Control the speed of Space/Play button.
• Select canvas background color. You can choose a background color or enter manually (in RGB or HEX format).
• Reset zoom Show every image in full size or zoomed out like the previous (it is enabled by default for interpolation mode and disabled for annotation mode).
• Rotate all images checkbox — switch the rotation of all frames or an individual frame.
• Smooth image checkbox — smooth image when zoom-in it.

  smoothed	pixelized

Workspace

In the Workspace tab, you can:

• Enable auto save checkbox — turned off by default.

• Auto save interval (min) input box — 15 minutes by default.

• Show all interpolation tracks checkbox — shows hidden objects on the side panel for every interpolated object (turned off by default).

• Always show object details - show text for an object on the canvas not only when the object is activated:

  

• Content of a text - setup of the composition of the object details:

  • ID - object identifier.
  • Attributes - attributes of the object.
  • Label - object label.
  • Source- source of creating of objects MANUAL, AUTO or SEMI-AUTO.
  • Descriptions - description of attributes.
  • Dimensions - width, height and rotation for rectangles and ellipses.

• Position of a text - text positioning mode selection:

  • Auto - the object details will be automatically placed where free space is.
  • Center - the object details will be embedded to a corresponding object if possible.

• Font size of a text - specifies the text size of the object details.

• Automatic bordering - enable automatic bordering for polygons and polylines during drawing/editing. For more information To find out more, go to the section annotation with polygons.

• Intelligent polygon cropping - activates intelligent cropping when editing the polygon (read more in the section edit polygon

• Show tags on frame - shows/hides frame tags on the current frame

• Attribute annotation mode (AAM) zoom margin input box — defines margins (in px) for shape in the attribute annotation mode.

• Control points size — defines the size of any interactable points in the tool (polygon’s vertices, rectangle dragging points, etc.)

• Default number of points in polygon approximation With this setting, you can choose the default number of points in polygon. Works for serverless interactors and OpenCV scissors.

• Select Save to save settings (settings will be saved on the server and will not change after the page is refreshed). Select Cancel or press F2 to return to the annotation.

Shortcuts

In the Shortcuts tab, you can set the keyboard combinations. Learn more in Shortcuts.

1.6 - 3D task workspace

If the related_images folder contains any images, a context image will be available in the perspective window. The contextual image could be compared to 3D data and would help to identify the labels of marked objects.

Perspective – a main window for work with objects in a 3D task.

Projections - projections are tied to an object so that a cuboid is in the center and looks like a rectangle. Projections show only the selected object.

• Top – a projection of the view from above.
• Side – a projection of the left side of the object.
• Front - a frontal projection of the object.

2 - Tools & Shapes

2.1 - Shape mode (advanced)

Basic operations in the mode were described in section shape mode (basics).

Occluded Occlusion is an attribute used if an object is occluded by another object or isn’t fully visible on the frame. Use Q shortcut to set the property quickly.

Example: the three cars on the figure below should be labeled as occluded.

If a frame contains too many objects and it is difficult to annotate them due to many shapes placed mostly in the same place, it makes sense to lock them. Shapes for locked objects are transparent, and it is easy to annotate new objects. Besides, you can’t change previously annotated objects by accident. Shortcut: L.

2.2 - Single Shape

The CVAT Single Shape annotation mode accelerates the annotation process and enhances workflow efficiency for specific scenarios.

By using this mode you can label objects with a chosen annotation shape and label when an image contains only a single object. By eliminating the necessity to select tools from the sidebar and facilitating quicker navigation between images without the reliance on hotkeys, this feature makes the annotation process significantly faster.

See:

• Single Shape mode annotation interface
• Annotating in Single Shape mode
• Query parameters
• Video tutorial

Single Shape mode annotation interface

A set of controls in the interface of the Single Shape annotation mode may vary depending on different settings.

Images below displays the complete interface, featuring all available fields; as mentioned above, certain fields may be absent depending on the scenario.

For instance, when annotating with rectangles, the Number of points field will not appear, and if annotating a single class, the Labels selector will be omitted.

To access Single Shape mode, open the job, navigate to the top right corner, and from the drop-down menu, select Single Shape.

The interface will be different if the shape type was set to Any in the label Constructor:

The Single Shape annotation mode has the following fields:

Annotating in Single Shape mode

To annotate in Single Shape mode, follow these steps:

1. Open the job and switch to Single Shape mode.
2. Annotate the image based on the selected shape. For more information on shapes, see Annotation Tools.
3. (Optional) If the image does not contain any objects to annotate, click Skip at the top of the right panel.
4. Submit your work.

Query parameters

Also, we introduced additional query parameters, which you may append to the job link, to initialize the annotation process and automate workflow:

You can combine these parameters to customize the workspace for an annotator, for example:

/tasks/<tid>/jobs/<jid>?defaultWorkspace=single_shape&defaultLabel=dog&defaultPointsCount=10

Will open the following job:

Video tutorial

For a better understanding of how Single Shape mode operates, we recommend watching the following tutorial.

2.3 - Track mode (advanced)

Basic operations in the mode were described in section track mode (basics).

Shapes that were created in the track mode, have extra navigation buttons.

• These buttons help to jump to the previous/next keyframe.

  

• The button helps to jump to the initial frame and to the last keyframe.

  

You can use the Split function to split one track into two tracks:

2.4 - 3D Object annotation (advanced)

As well as 2D-task objects, 3D-task objects support the ability to change appearance, attributes, properties and have an action menu. Read more in objects sidebar section.

Moving an object

If you hover the cursor over a cuboid and press Shift+N, the cuboid will be cut, so you can paste it in other place (double-click to paste the cuboid).

Copying

As well as in 2D task you can copy and paste objects by Ctrl+C and Ctrl+V, but unlike 2D tasks you have to place a copied object in a 3D space (double click to paste).

Image of the projection window

You can copy or save the projection-window image by left-clicking on it and selecting a “save image as” or “copy image”.

Cuboid orientation

The feature enables or disables the display of cuboid orientation arrows in the 3D space. It is controlled by a checkbox located in the appearance block. When enabled, arrows representing the cuboid’s axis orientation (X - red, Y - green, Z - blue) are displayed, providing a visual reference for the cuboid’s alignment within the 3D environment. This feature is useful for understanding the spatial orientation of the cuboid.

Cuboid size input

The size input feature allows users to manually specify the dimensions of a cuboid in the 3D space. This feature is accessible through the objects sidebar - details panel, where you can input precise values for the width, height, and length (X - width, Y - height, Z - length) of the cuboid. By entering these values, the cuboid’s size is adjusted accordingly to its orientation, providing greater control and accuracy when annotating objects in 3D tasks.

2.5 - Attribute annotation mode (advanced)

Basic operations in the mode were described in section attribute annotation mode (basics).

It is possible to handle lots of objects on the same frame in the mode.

It is more convenient to annotate objects of the same type. In this case you can apply the appropriate filter. For example, the following filter will hide all objects except person: label=="Person".

To navigate between objects (person in this case), use the following buttons switch between objects in the frame on the special panel:

or shortcuts:

• Tab — go to the next object
• Shift+Tab — go to the previous object.

In order to change the zoom level, go to settings (press F3) in the workspace tab and set the value Attribute annotation mode (AAM) zoom margin in px.

2.6 - Annotation with rectangles

To learn more about annotation using a rectangle, see the sections:

• Shape mode (basics)
• Track mode (basics)
• Shape mode (advanced)
• Track mode (advanced)

Rotation rectangle

To rotate the rectangle, pull on the rotation point. Rotation is done around the center of the rectangle. To rotate at a fixed angle (multiple of 15 degrees), hold shift. In the process of rotation, you can see the angle of rotation.

Annotation with rectangle by 4 points

It is an efficient method of bounding box annotation, proposed here. Before starting, you need to make sure that the drawing method by 4 points is selected.

Press Shape or Track for entering drawing mode. Click on four extreme points: the top, bottom, left- and right-most physical points on the object. Drawing will be automatically completed right after clicking the fourth point. Press Esc to cancel editing.

2.7 - Annotation with polygons

2.7.1 - Manual drawing

It is used for semantic / instance segmentation.

Before starting, you need to select Polygon on the controls sidebar and choose the correct Label.

• Click Shape to enter drawing mode. There are two ways to draw a polygon: either create points by clicking or by dragging the mouse on the screen while holding Shift.

Clicking points	Holding Shift+Dragging

• When Shift isn’t pressed, you can zoom in/out (when scrolling the mouse wheel) and move (when clicking the mouse wheel and moving the mouse), you can also delete the previous point by right-clicking on it.
• You can use the Selected opacity slider in the Objects sidebar to change the opacity of the polygon. You can read more in the Objects sidebar section.
• Press N again or click the Done button on the top panel for completing the shape.
• After creating the polygon, you can move the points or delete them by right-clicking and selecting Delete point or clicking with pressed Alt key in the context menu.

2.7.2 - Drawing using automatic borders

You can use auto borders when drawing a polygon. Using automatic borders allows you to automatically trace the outline of polygons existing in the annotation.

• To do this, go to settings -> workspace tab and enable Automatic Bordering or press Ctrl while drawing a polygon.

  

• Start drawing / editing a polygon.

• Points of other shapes will be highlighted, which means that the polygon can be attached to them.

• Define the part of the polygon path that you want to repeat.

  

• Click on the first point of the contour part.

  

• Then click on any point located on part of the path. The selected point will be highlighted in purple.

  

• Click on the last point and the outline to this point will be built automatically.

  

Besides, you can set a fixed number of points in the Number of points field, then drawing will be stopped automatically. To enable dragging you should right-click inside the polygon and choose Switch pinned property.

Below you can see results with opacity and black stroke:

If you need to annotate small objects, increase Image Quality to 95 in Create task dialog for your convenience.

2.7.3 - Edit polygon

To edit a polygon you have to click on it while holding Shift, it will open the polygon editor.

• In the editor you can create new points or delete part of a polygon by closing the line on another point.

• When Intelligent polygon cropping option is activated in the settings, CVAT considers two criteria to decide which part of a polygon should be cut off during automatic editing.

  • The first criteria is a number of cut points.
  • The second criteria is a length of a cut curve.

  If both criteria recommend to cut the same part, algorithm works automatically, and if not, a user has to make the decision. If you want to choose manually which part of a polygon should be cut off, disable Intelligent polygon cropping in the settings. In this case after closing the polygon, you can select the part of the polygon you want to leave.

  

• You can press Esc to cancel editing.

  

2.7.4 - Track mode with polygons

Polygons in the track mode allow you to mark moving objects more accurately other than using a rectangle (Tracking mode (basic); Tracking mode (advanced)).

1. To create a polygon in the track mode, click the Track button.

   

2. Create a polygon the same way as in the case of Annotation with polygons. Press N or click the Done button on the top panel to complete the polygon.

3. Pay attention to the fact that the created polygon has a starting point and a direction, these elements are important for annotation of the following frames.

4. After going a few frames forward press Shift+N, the old polygon will disappear and you can create a new polygon. The new starting point should match the starting point of the previously created polygon (in this example, the top of the left mirror). The direction must also match (in this example, clockwise). After creating the polygon, press N and the intermediate frames will be interpolated automatically.

   

5. If you need to change the starting point, right-click on the desired point and select Set starting point. To change the direction, right-click on the desired point and select switch orientation.

   

There is no need to redraw the polygon every time using Shift+N, instead you can simply move the points or edit a part of the polygon by pressing Shift+Click.

2.7.5 - Creating masks

Cutting holes in polygons

Currently, CVAT does not support cutting transparent holes in polygons. However, it is possible to generate holes in exported instance and class masks. To do this, one needs to define a background class in the task and draw holes with it as additional shapes above the shapes needed to have holes:

The editor window:

Remember to use z-axis ordering for shapes by [-] and [+, =] keys.

Exported masks:

Notice that it is currently impossible to have a single instance number for internal shapes (they will be merged into the largest one and then covered by “holes”).

Creating masks

There are several formats in CVAT that can be used to export masks:

• Segmentation Mask (PASCAL VOC masks)
• CamVid
• MOTS
• ICDAR
• COCO (RLE-encoded instance masks, guide)
• Datumaro

An example of exported masks (in the Segmentation Mask format):

Important notices:

• Both boxes and polygons are converted into masks
• Grouped objects are considered as a single instance and exported as a single mask (label and attributes are taken from the largest object in the group)

Class colors

All the labels have associated colors, which are used in the generated masks. These colors can be changed in the task label properties:

Label colors are also displayed in the annotation window on the right panel, where you can show or hide specific labels (only the presented labels are displayed):

A background class can be:

• A default class, which is implicitly-added, of black color (RGB 0, 0, 0)
• background class with any color (has a priority, name is case-insensitive)
• Any class of black color (RGB 0, 0, 0)

To change background color in generated masks (default is black), change background class color to the desired one.

2.8 - Annotation with polylines

It is used for road markup annotation etc.

Before starting, you need to select the Polyline. You can set a fixed number of points in the Number of points field, then drawing will be stopped automatically.

Click Shape to enter drawing mode. There are two ways to draw a polyline — you either create points by clicking or by dragging a mouse on the screen while holding Shift. When Shift isn’t pressed, you can zoom in/out (when scrolling the mouse wheel) and move (when clicking the mouse wheel and moving the mouse), you can delete previous points by right-clicking on it. Press N again or click the Done button on the top panel to complete the shape. You can delete a point by clicking on it with pressed Ctrl or right-clicking on a point and selecting Delete point. Click with pressed Shift will open a polyline editor. There you can create new points(by clicking or dragging) or delete part of a polygon closing the red line on another point. Press Esc to cancel editing.

2.9 - Annotation with points

2.9.1 - Points in shape mode

It is used for face, landmarks annotation etc.

Before you start you need to select the Points. If necessary you can set a fixed number of points in the Number of points field, then drawing will be stopped automatically.

Click Shape to entering the drawing mode. Now you can start annotation of the necessary area. Points are automatically grouped — all points will be considered linked between each start and finish. Press N again or click the Done button on the top panel to finish marking the area. You can delete a point by clicking with pressed Ctrl or right-clicking on a point and selecting Delete point. Clicking with pressed Shift will open the points shape editor. There you can add new points into an existing shape. You can zoom in/out (when scrolling the mouse wheel) and move (when clicking the mouse wheel and moving the mouse) while drawing. You can drag an object after it has been drawn and change the position of individual points after finishing an object.

2.9.2 - Linear interpolation with one point

You can use linear interpolation for points to annotate a moving object:

1. Before you start, select the Points.

2. Linear interpolation works only with one point, so you need to set Number of points to 1.

3. After that select the Track.

   

4. Click Track to enter the drawing mode left-click to create a point and after that shape will be automatically completed.

   

5. Move forward a few frames and move the point to the desired position, this way you will create a keyframe and intermediate frames will be drawn automatically. You can work with this object as with an interpolated track: you can hide it using the Outside, move around keyframes, etc.

   

6. This way you’ll get linear interpolation using the Points.

   

2.10 - Annotation with ellipses

It is used for road sign annotation etc.

First of all you need to select the ellipse on the controls sidebar.

Choose a Label and click Shape or Track to start drawing. An ellipse can be created the same way as a rectangle, you need to specify two opposite points, and the ellipse will be inscribed in an imaginary rectangle. Press N or click the Done button on the top panel to complete the shape.

You can rotate ellipses using a rotation point in the same way as rectangles.

Annotation with ellipses video tutorial

2.11 - Annotation with cuboids

It is used to annotate 3 dimensional objects such as cars, boxes, etc… Currently the feature supports one point perspective and has the constraint where the vertical edges are exactly parallel to the sides.

2.11.1 - Creating the cuboid

Before you start, you have to make sure that Cuboid is selected and choose a drawing method from rectangle or by 4 points.

Drawing cuboid by 4 points

Choose a drawing method by 4 points and select Shape to enter the drawing mode. There are many ways to draw a cuboid. You can draw the cuboid by placing 4 points, after that the drawing will be completed automatically. The first 3 points determine the plane of the cuboid while the last point determines the depth of that plane. For the first 3 points, it is recommended to only draw the 2 closest side faces, as well as the top and bottom face.

A few examples:

Drawing cuboid from rectangle

Choose a drawing method from rectangle and select Shape to enter the drawing mode. When you draw using the rectangle method, you must select the frontal plane of the object using the bounding box. The depth and perspective of the resulting cuboid can be edited.

Example:

2.11.2 - Editing the cuboid

The cuboid can be edited in multiple ways: by dragging points, by dragging certain faces or by dragging planes. First notice that there is a face that is painted with gray lines only, let us call it the front face.

You can move the cuboid by simply dragging the shape behind the front face. The cuboid can be extended by dragging on the point in the middle of the edges. The cuboid can also be extended up and down by dragging the point at the vertices.

To draw with perspective effects it should be assumed that the front face is the closest to the camera. To begin simply drag the points on the vertices that are not on the gray/front face while holding Shift. The cuboid can then be edited as usual.

If you wish to reset perspective effects, you may right click on the cuboid, and select Reset perspective to return to a regular cuboid.

The location of the gray face can be swapped with the adjacent visible side face. You can do it by right clicking on the cuboid and selecting Switch perspective orientation. Note that this will also reset the perspective effects.

Certain faces of the cuboid can also be edited, these faces are: the left, right and dorsal faces, relative to the gray face. Simply drag the faces to move them independently from the rest of the cuboid.

You can also use cuboids in track mode, similar to rectangles in track mode (basics and advanced) or Track mode with polygons.

2.12 - Annotation with skeletons

In this guide, we delve into the efficient process of annotating complex structures through the implementation of Skeleton annotations.

Skeletons serve as annotation templates for annotating complex objects with a consistent structure, such as human pose estimation or facial landmarks.

A Skeleton is composed of numerous points (also referred to as elements), which may be connected by edges. Each point functions as an individual object, possessing unique attributes and properties like color, occlusion, and visibility.

Skeletons can be exported in two formats: CVAT for image and COCO Keypoints.

See:

• Adding Skeleton manually
  • Skeleton Configurator
  • Configuring Skeleton points
  • Adding Skeleton labels manually
• Adding Skeleton labels from the model
• Annotation with Skeletons
• Automatic annotation with Skeletons
• Editing skeletons on the canvas
• Editing skeletons on the sidebar

Adding Skeleton manually

To start annotating using skeletons, you need to set up a Skeleton task in Configurator:

To open Configurator, when creating a task, click on the Setup skeleton button if you want to set up the skeleton manually, or From model if you want to add skeleton labels from a model.

Skeleton Configurator

The skeleton Configurator is a tool to build skeletons for annotation. It has the following fields:

Configuring Skeleton points

You can name labels, set attributes, and change the color of each point of the skeleton.

To do this, right-click on the skeleton point and select Configure:

In the opened menu, you can change the point setting. It is similar to adding labels and attributes of the regular task:

A Skeleton point can only exist within its parent Skeleton.

Adding Skeleton labels manually

To create the Skeleton task, do the following:

1. Open Configurator.
2. (Optional) Upload background image.
3. In the Label name field, enter the name of the label.
4. (Optional) Add attribute
   Note: you can add attributes exclusively to each point, for more information, see Configuring Skeleton points
5. Use Add point to add points to the Drawing area.
6. Use Add edge to add edges between points.
7. Upload files.
8. Click:
   • Submit & Open to create and open the task.
   • Submit & Continue to submit the configuration and start creating a new task.

Adding Skeleton labels from the model

To add points from the model, and annotate do the following:

1. Open Basic configurator.

2. On the Constructor tab, click From model.

3. From the Select a model to pick labels select the Human pose estimation model or others if available.

4. Click on the model’s labels, you want to use.
   Selected labels will become gray.

   

5. (Optional) If you want to adjust labels, within the label, click the Update attributes icon.
   The Skeleton configurator will open, where you can configure the skeleton.
   Note: Labels cannot be adjusted after the task/project is created.

6. Click Done. The labels, that you selected, will appear in the labels window.

7. Upload data.

8. Click:

   • Submit & Open to create and open the task.
   • Submit & Continue to submit the configuration and start creating a new task.

Annotation with Skeletons

To annotate with Skeleton, do the following

1. Open job.

2. On the tools panel select Draw new skeleton.

3. Select Track to annotate with tracking or Shape to annotate without tracking.

   

4. Draw a skeleton on the image.

Automatic annotation with Skeletons

To automatically annotate with Skeleton, do the following

1. Open the job and on the tools panel select AI Tools > Detectors

2. From the drop-down list select the model. You will see a list of points to match and the name of the skeleton on the top of the list.

   

3. (Optional) By clicking on the Bin icon, you can remove any mapped item:

   • A skeleton together with all points.
   • Certain points from two mapped skeletons.

4. Click Annotate.

Editing skeletons on the canvas

A drawn skeleton is encompassed within a bounding box, it allows you to manipulate the skeleton as a regular bounding box, enabling actions such as dragging, resizing, or rotating:

Upon repositioning a point, the bounding box adjusts automatically, without affecting other points:

Additionally, Shortcuts are applicable to both the skeleton as a whole and its elements:

• To use a shortcut to the entire skeleton, hover over the bounding box and push the shortcut keyboard key. This action is applicable for shortcuts like the lock, occluded, pinned, keyframe, and outside for skeleton tracks.
• To use a shortcut to a specific skeleton point, hover over the point and push the shortcut keyboard key. The same list of shortcuts is available, with the addition of outside, which is also applicable to individual skeleton shape elements.

Editing skeletons on the sidebar

In CVAT, the sidebar offers an alternative method for setting up skeleton properties and attributes.

This approach is similar to that used for other object types supported by CVAT, but with a few specific alterations:

An additional collapsible section is provided for users to view a comprehensive list of skeleton parts.

Skeleton points can have properties like Outside, Occluded, and Hidden.

Both Outside and Hidden make a skeleton point invisible.

• Outside property is part of annotations. Use it when part of the object is out of frame borders.

• Hidden makes a point hidden only for the annotator’s convenience, this property will not be saved between different sessions.

• Occluded keeps the point visible on the frame and usually means that the point is still on a frame, just hidden behind another object.

2.13 - Annotation with brush tool

With a brush tool, you can create masks for disjoint objects, that have multiple parts, such as a house hiding behind trees, a car behind a pedestrian, or a pillar behind a traffic sign. The brush tool has several modes, for example: erase pixels, change brush shapes, and polygon-to-mask mode.

Use brush tool for Semantic (Panoptic) and Instance Image Segmentation tasks.
For more information about segmentation masks in CVAT, see Creating masks.

See:

• Brush tool menu
• Annotation with brush
• Annotation with polygon-to-mask
• Remove underlying pixels
• AI Tools
• Import and export

Brush tool menu

The brush tool menu appears on the top of the screen after you click Shape:

It has the following elements:

Element	Description
	Save mask saves the created mask. The saved mask will appear on the object sidebar
	Save mask and continue adds a new mask to the object sidebar and allows you to draw a new one immediately.
	Brush adds new mask/ new regions to the previously added mask).
	Eraser removes part of the mask.
	Polygon selection tool. Selection will become a mask.
	Remove polygon selection subtracts part of the polygon selection.
	Brush size in pixels. Note: Visible only when Brush or Eraser are selected.
	Brush shape with two options: circle and square. Note: Visible only when Brush or Eraser are selected.
	Remove underlying pixels. When you are drawing or editing a mask with this tool, pixels on other masks that are located at the same positions as the pixels of the current mask are deleted.
	Hide mask. When drawing or editing a mask, you can enable this feature to temporarily hide the mask, allowing you to see the objects underneath more clearly.
	Label that will be assigned to the newly created mask
	Move. Click and hold to move the menu bar to the other place on the screen

Annotation with brush

To annotate with brush, do the following:

1. From the controls sidebar, select Brush .

2. In the Draw new mask menu, select label for your mask, and click Shape.
   The Brush tool will be selected by default.

   

3. With the brush, draw a mask on the object you want to label.
   To erase selection, use Eraser

   

4. After you applied the mask, on the top menu bar click Save mask
   to finish the process (or N on the keyboard).

5. Added object will appear on the objects sidebar.

To add the next object, repeat steps 1 to 5. All added objects will be visible on the image and the objects sidebar.

To save the job with all added objects, on the top menu, click Save .

Annotation with polygon-to-mask

To annotate with polygon-to-mask, do the following:

1. From the controls sidebar, select Brush .

2. In the Draw new mask menu, select label for your mask, and click Shape.

   

3. In the brush tool menu, select Polygon .

4. With the Polygon tool, draw a mask for the object you want to label.
   To correct selection, use Remove polygon selection .

5. Use Save mask (or N on the keyboard)
   to switch between add/remove polygon tools:

   

6. After you added the polygon selection, on the top menu bar click Save mask
   to finish the process (or N on the keyboard).

7. Click Save mask again (or N on the keyboard).
   The added object will appear on the objects sidebar.

To add the next object, repeat steps 1 to 5.

All added objects will be visible on the image and the objects sidebar.

To save the job with all added objects, on the top menu, click Save .

Remove underlying pixels

Use Remove underlying pixels tool when you want to add a mask and simultaneously delete the pixels of
other masks that are located at the same positions. It is a highly useful feature to avoid meticulous drawing edges twice between two different objects.

AI Tools

You can convert AI tool masks to polygons. To do this, use the following AI tool menu:

1. Go to the Detectors tab.
2. Switch toggle Masks to polygons to the right.
3. Add source and destination labels from the drop-down lists.
4. Click Annotate.

Import and export

For export, see Export dataset

Import follows the general import dataset procedure, with the additional option of converting masks to polygons.

To use it, when uploading the dataset, switch the Convert masks to polygon toggle to the right:

2.14 - Annotation with tags

It is used to annotate frames, tags are not displayed in the workspace. Before you start, open the drop-down list in the top panel and select Tag annotation.

The objects sidebar will be replaced with a special panel for working with tags. Here you can select a label for a tag and add it by clicking on the Plus button. You can also customize hotkeys for each label.

If you need to use only one label for one frame, then enable the Automatically go to the next frame checkbox, then after you add the tag the frame will automatically switch to the next.

Tags will be shown in the top left corner of the canvas. You can show/hide them in the settings.

2.15 - AI Tools

Label and annotate your data in semi-automatic and automatic mode with the help of AI and OpenCV tools.

While interpolation is good for annotation of the videos made by the security cameras, AI and OpenCV tools are good for both: videos where the camera is stable and videos, where it moves together with the object, or movements of the object are chaotic.

See:

• Interactors
  • AI tools: annotate with interactors
  • AI tools: add extra points
  • AI tools: delete points
  • OpenCV: intelligent scissors
  • Settings
  • Interactors models
• Detectors
  • Labels matching
  • Annotate with detectors
  • Detectors models
• Trackers
  • AI tools: annotate with trackers
  • OpenCV: annotate with trackers
  • When tracking
  • Trackers models
• OpenCV: histogram equalization

Interactors

Interactors are a part of AI and OpenCV tools.

Use interactors to label objects in images by creating a polygon semi-automatically.

When creating a polygon, you can use positive points or negative points (for some models):

• Positive points define the area in which the object is located.
• Negative points define the area in which the object is not located.

AI tools: annotate with interactors

To annotate with interactors, do the following:

1. Click Magic wand , and go to the Interactors tab.
2. From the Label drop-down, select a label for the polygon.
3. From the Interactor drop-down, select a model (see Interactors models).
   Click the Question mark to see information about each model:
   
4. (Optional) If the model returns masks, and you need to convert masks to polygons, use the Convert masks to polygons toggle.
5. Click Interact.
6. Use the left click to add positive points and the right click to add negative points.
   Number of points you can add depends on the model.
7. On the top menu, click Done (or Shift+N, N).

AI tools: add extra points

Each model has a minimum required number of points for annotation. Once the required number of points is reached, the request is automatically sent to the server. The server processes the request and adds a polygon to the frame.

For a more accurate outline, postpone request to finish adding extra points first:

1. Hold down the Ctrl key.
   On the top panel, the Block button will turn blue.
2. Add points to the image.
3. Release the Ctrl key, when ready.

In case you used Mask to polygon when the object is finished, you can edit it like a polygon.

You can change the number of points in the polygon with the slider:

AI tools: delete points

To delete a point, do the following:

1. With the cursor, hover over the point you want to delete.
2. If the point can be deleted, it will enlarge and the cursor will turn into a cross.
3. Left-click on the point.

OpenCV: intelligent scissors

To use Intelligent scissors, do the following:

1. On the menu toolbar, click OpenCV and wait for the library to load.

   
   

2. Go to the Drawing tab, select the label, and click on the Intelligent scissors button.

   

3. Add the first point on the boundary of the allocated object.
   You will see a line repeating the outline of the object.

4. Add the second point, so that the previous point is within the restrictive threshold.
   After that a line repeating the object boundary will be automatically created between the points.

5. To finish placing points, on the top menu click Done (or N on the keyboard).

As a result, a polygon will be created.

You can change the number of points in the polygon with the slider:

To increase or lower the action threshold, hold Ctrl and scroll the mouse wheel.

During the drawing process, you can remove the last point by clicking on it with the left mouse button.

Settings

• On how to adjust the polygon, see Objects sidebar.

• For more information about polygons in general, see Annotation with polygons.

Interactors models

Model	Tool	Description	Example
Segment Anything Model (SAM)	AI Tools	The Segment Anything Model (SAM) produces high quality object masks, and it can be used to generate masks for all objects in an image. It has been trained on a dataset of 11 million images and 1.1 billion masks, and has strong zero-shot performance on a variety of segmentation tasks. For more information, see: GitHub: Segment Anything Site: Segment Anything Paper: Segment Anything
Deep extreme cut (DEXTR)	AI Tool	This is an optimized version of the original model, introduced at the end of 2017. It uses the information about extreme points of an object to get its mask. The mask is then converted to a polygon. For now this is the fastest interactor on the CPU. For more information, see: GitHub: DEXTR-PyTorch Site: DEXTR-PyTorch Paper: DEXTR-PyTorch
Inside-Outside-Guidance (IOG)	AI Tool	The model uses a bounding box and inside/outside points to create a mask. First of all, you need to create a bounding box, wrapping the object. Then you need to use positive and negative points to say the model where is a foreground, and where is a background. Negative points are optional. For more information, see: GitHub: IOG Paper: IOG
Intelligent scissors	OpenCV	Intelligent scissors is a CV method of creating a polygon by placing points with the automatic drawing of a line between them. The distance between the adjacent points is limited by the threshold of action, displayed as a red square that is tied to the cursor. For more information, see: Site: Intelligent Scissors Specification

Detectors

Detectors are a part of AI tools.

Use detectors to automatically identify and locate objects in images or videos.

Labels matching

Each model is trained on a dataset and supports only the dataset’s labels.

For example:

• DL model has the label car.
• Your task (or project) has the label vehicle.

To annotate, you need to match these two labels to give DL model a hint, that in this case car = vehicle.

If you have a label that is not on the list of DL labels, you will not be able to match them.

For this reason, supported DL models are suitable only for certain labels.
To check the list of labels for each model, see Detectors models.

Annotate with detectors

To annotate with detectors, do the following:

1. Click Magic wand , and go to the Detectors tab.

2. From the Model drop-down, select model (see Detectors models).

3. From the left drop-down select the DL model label, from the right drop-down select the matching label of your task.

   

4. (Optional) If the model returns masks, and you need to convert masks to polygons, use the Convert masks to polygons toggle.

5. (Optional) You can specify a Threshold for the model. If not provided, the default value from the model settings will be used.

6. Click Annotate.

This action will automatically annotate one frame. For automatic annotation of multiple frames, see Automatic annotation.

Detectors models

Trackers

Trackers are part of AI and OpenCV tools.

Use trackers to identify and label objects in a video or image sequence that are moving or changing over time.

AI tools: annotate with trackers

To annotate with trackers, do the following:

1. Click Magic wand , and go to the Trackers tab.

   
   

2. From the Label drop-down, select the label for the object.

3. From Tracker drop-down, select tracker.

4. Click Track, and annotate the objects with the bounding box in the first frame.

5. Go to the top menu and click Next (or the F on the keyboard) to move to the next frame.
   All annotated objects will be automatically tracked.

When tracking

• To enable/disable tracking, use Tracker switcher on the sidebar.

  

• Trackable objects have an indication on canvas with a model name.

  

• You can follow the tracking by the messages appearing at the top.

  

OpenCV: annotate with trackers

To annotate with trackers, do the following:

1. Create basic rectangle shapes or tracks for tracker initialization

2. On the menu toolbar, click OpenCV and wait for the library to load.

   
   

3. From Tracker drop-down, select tracker and Click Track

   
   

4. Annotation actions window will pop-up. Setup Target frame and Convert rectangle shapes to tracks parameters and click Run

   Note

   Tracking will be applied to all filtered rectangle annotations.

   
   

All annotated objects will be automatically tracked up until target frame parameter.

Trackers models

Model	Tool	Description	Example
TrackerMIL	OpenCV	TrackerMIL model is not bound to labels and can be used for any object. It is a fast client-side model designed to track simple non-overlapping objects. For more information, see: Article: Object Tracking using OpenCV
SiamMask	AI Tools	Fast online Object Tracking and Segmentation. The trackable object will be tracked automatically if the previous frame was the latest keyframe for the object. For more information, see: GitHub: SiamMask Paper: SiamMask
Transformer Tracking (TransT)	AI Tools	Simple and efficient online tool for object tracking and segmentation. If the previous frame was the latest keyframe for the object, the trackable object will be tracked automatically. This is a modified version of the PyTracking Python framework based on Pytorch For more information, see: GitHub: TransT Paper: TransT
SAM2 Tracker	AI Agent	Advanced object tracking and segmentation using Meta’s Segment Anything Model 2. Available for CVAT Online and Enterprise via AI agents. Supports polygons and masks with high precision tracking. Requires user-side agent setup with Python 3.10+. For more information, see: SAM2 Tracker Setup Guide SAM2 Blog: AI Agent Integration	Example coming soon

OpenCV: histogram equalization

Histogram equalization improves the contrast by stretching the intensity range.

It increases the global contrast of images when its usable data is represented by close contrast values.

It is useful in images with backgrounds and foregrounds that are bright or dark.

To improve the contrast of the image, do the following:

1. In the OpenCV menu, go to the Image tab.
2. Click on Histogram equalization button.
   

Histogram equalization will improve contrast on current and following frames.

Example of the result:

To disable Histogram equalization, click on the button again.

2.16 - Standard 3D mode (basics)

Standard 3d mode - Designed to work with 3D data. The mode is automatically available if you add PCD or Kitty BIN format data when you create a task (read more).

You can adjust the size of the projections, to do so, simply drag the boundary between the projections.

2.17 - Types of shapes

There are several shapes with which you can annotate your images:

• Rectangle or Bounding box
• Polygon
• Polyline
• Points
• Ellipse
• Cuboid
• Cuboid in 3d task
• Skeleton
• Tag

And there is what they look like:

Tag - has no shape in the workspace, but is displayed in objects sidebar.

2.18 - Backup Task and Project

Overview

In CVAT you can backup tasks and projects. This can be used to backup a task or project on your PC or to transfer to another server.

Create backup

To backup a task or project, open the action menu and select Backup Task or Backup Project.

You can backup a project or a task locally on your PC or using an attached cloud storage.

The dialog includes a switch Use lightweight backup whenever possible. When enabled, CVAT creates a lightweight backup for data that comes from attached cloud storage: the backup stores task/project meta and annotations and does not copy raw media files. This reduces backup size and time for cloud-backed data. The switch has no effect on a tasks, whose data is located on CVAT. The switch is enabled by default.

(Optional) Specify the name in the Custom name text field for backup, otherwise the file of backup name will be given by the mask project_<project_name>_backup_<date>_<time>.zip for the projects and task_<task_name>_backup_<date>_<time>.zip for the tasks.

If you want to save a backup to a specific attached cloud storage, you should additionally turn off the switch Use default settings, select the Cloud storage value in the Target storage and select this storage in the list of the attached cloud storages.

Create backup APIs

• endpoints:
  • /tasks/{id}/backup
  • /projects/{id}/backup
• method: GET
• responses: 202, 201 with zip archive payload

Upload backup APIs

• endpoints:
  • /api/tasks/backup
  • /api/projects/backup
• method: POST
• Content-Type: multipart/form-data
• responses: 202, 201 with json payload

Create from backup

To create a task or project from a backup, go to the tasks or projects page, click the Create from backup button and select the archive you need.

As a result, you’ll get a task containing data, parameters, and annotations of the previously exported task.

Note: When restoring from a lightweight backup, CVAT creates a task which is not attached to cloud storage. Data cannot be fetched until cloud storage is attached on a Task page.

Backup file structure

As a result, you’ll get a zip archive containing data, task or project and task specification and annotations with the following structure:

• Task Backup Structure
• Project Backup Structure

    .
    ├── data
    │   └── {user uploaded data}
    ├── task.json
    └── annotations.json
  

    .
    ├── task_{id}
    │   ├── data
    │   │   └── {user uploaded data}
    │   ├── task.json
    │   └── annotations.json
    └── project.json
  

2.19 - Shape mode (basics)

Usage examples:

• Create new annotations for a set of images.
• Add/modify/delete objects for existing annotations.

1. You need to select Rectangle on the controls sidebar:

   

   Before you start, select the correct Label (should be specified by you when creating the task) and Drawing Method (by 2 points or by 4 points):

   

2. Creating a new annotation in Shape mode:

   • Create a separate Rectangle by selecting Shape.

     

   • Choose the opposite points. Your first rectangle is ready!

     

   • To learn more about creating a rectangle read here.

   • It is possible to adjust boundaries and location of the rectangle using a mouse. The rectangle’s size is shown in the top right corner, you can check it by selecting any point of the shape. You can also undo your actions using Ctrl+Z and redo them with Shift+Ctrl+Z or Ctrl+Y.

3. You can see the Object card in the objects sidebar or open it by right-clicking on the object. You can change the attributes in the details section. You can perform basic operations or delete an object by selecting on the action menu button.

   

4. The following figure is an example of a fully annotated frame with separate shapes.

   

   Read more in the section shape mode (advanced).

2.20 - Frame deleting

Delete frame

You can delete the current frame from a task. This frame will not be presented either in the UI or in the exported annotation. Thus, it is possible to mark corrupted frames that are not subject to annotation.

1. Go to the Job annotation view and click on the Delete frame button (Alt+Del).

   Note

   When you delete with the shortcut, the frame will be deleted immediately without additional confirmation.

   

2. After that you will be asked to confirm frame deleting.

   Note

   all annotations from that frame will be deleted, unsaved annotations will be saved and the frame will be invisible in the annotation view (Until you make it visible in the settings). If there is some overlap in the task and the deleted frame falls within this interval, then this will cause this frame to become unavailable in another job as well.

3. When you delete a frame in a job with tracks, you may need to adjust some tracks manually. Common adjustments are:

   • Add keyframes at the edges of the deleted interval for the interpolation to look correct;
   • Move the keyframe start or end keyframe to the correct side of the deleted interval.

Configure deleted frames visibility and navigation

If you need to enable showing the deleted frames, you can do it in the settings.

1. Go to the settings and chose Player settings.

   

2. Click on the Show deleted frames checkbox. And close the settings dialog.

   

3. Then you will be able to navigate through deleted frames. But annotation tools will be unavailable. Deleted frames differ in the corresponding overlay.

4. There are ways to navigate through deleted frames without enabling this option:

   • Go to the frame via direct navigation methods: navigation slider or frame input field,
   • Go to the frame via the direct link, for example: /api/tasks/{id}/jobs/{id}?frame={frame_id}.

5. Navigation with step will not count deleted frames.

Restore deleted frame

You can also restore deleted frames in the task.

1. Turn on deleted frames visibility, as it was told in the previous part, and go to the deleted frame you want to restore.

   

2. Click on the Restore icon. The frame will be restored immediately.

2.21 - Join and slice tools

In CVAT you can modify shapes by either joining multiple shapes into a single label or slicing a single label into several shapes.

This document provides guidance on how to perform these operations effectively.

See:

• Joining masks
• Slicing polygons and masks

Joining masks

The Join masks tool (), is specifically designed to work with mask annotations.

This tool is useful in scenarios where a single object in an image is annotated with multiple shapes, and there is a need to merge these shapes into a single one.

To join masks, do the following:

1. From the Edit block, select Join masks .
2. Click on the canvas area, to select masks that you want to join.
3. (Optional) To remove the selection click the mask one more time.
4. Click again on Join masks (J) to execute the join operation.

Upon completion, the selected masks will be joined into a single mask.

Slicing polygons and masks

The Slice mask/polygon () is compatible with both mask and polygon annotations.

This tool is useful in scenarios where multiple objects in an image are annotated with one shape, and there is a need to slice this shape into multiple parts.

To slice mask or polygon, do the following:

1. From the Edit block, select Slice mask/polygon .
2. Click on the shape you intend to slice. A black contour will appear around the selected shape.
3. Set an initial point for slicing by clicking on the contour.
4. Draw a line across the shape to define the slicing path.
   Hold Shift to add points automatically on cursor movement.
   Note: The line cannot cross itself.
   Note: The line cannot cross the contour more than twice.
5. (Optional)> Right-click to cancel the latest point.
6. Click on the contour (Alt+J) (outside the contour) to finalize the slicing.

2.22 - Track mode (basics)

Usage examples:

• Create new annotations for a sequence of frames.
• Add/modify/delete objects for existing annotations.
• Edit tracks, merge several rectangles into one track.

1. Like in the Shape mode, you need to select a Rectangle on the sidebar, in the appearing form, select the desired Label and the Drawing method.

   

2. Creating a track for an object (look at the selected car as an example):

   • Create a Rectangle in Track mode by selecting Track.

     

   • In Track mode, the rectangle will be automatically interpolated on the next frames.

   • The cyclist starts moving on frame #2270. Let’s mark the frame as a key frame. You can press K for that or select the star button (see the screenshot below).

     

   • If the object starts to change its position, you need to modify the rectangle where it happens. It isn’t necessary to change the rectangle on each frame, simply update several keyframes and the frames between them will be interpolated automatically.

   • Let’s jump 30 frames forward and adjust the boundaries of the object. See an example below:

     

   • After that the rectangle of the object will be changed automatically on frames 2270 to 2300:

     

3. When the annotated object disappears or becomes too small, you need to finish the track. You have to choose Outside Property, shortcut O.

   

4. If the object isn’t visible on a couple of frames and then appears again, you can use the Merge feature to merge several individual tracks into one.

   

   • Create tracks for moments when the cyclist is visible:

     

   • Select Merge button or press key M and select on any rectangle of the first track and on any rectangle of the second track and so on:

     

   • Select Merge button or press M to apply changes.

     

   • The final annotated sequence of frames in Interpolation mode can look like the clip below:

     

     Read more in the section track mode (advanced).

2.23 - 3D Object annotation

Use the 3D Annotation tool for labeling 3D objects and scenes, such as vehicles, buildings, landscapes, and others.

Check out:

• Navigation
• Annotation with cuboids
  • Annotation with shapes
  • Tracking with cuboids

Navigation

The 3D annotation canvas looks like the following:

For information on the available tools, consult Controls sidebar.

You can navigate, using the mouse, or navigation keys:

You can also use keyboard shortcuts to navigate:

Annotation with cuboids

There are two options available for 3D annotation:

• Shape: for tasks like object detection.
• Track: uses interpolation to predict the position of objects in subsequent frames. A unique ID will be assigned to each object and maintained throughout the sequence of images.

Annotation with shapes

To add a 3D shape:

1. On the objects pane, select Draw new cuboid > select the label from the drop-down list > Shape.

   

2. The cursor will be followed by a cuboid. Place the cuboid on the 3D scene.

   

3. Use projections to adjust the cuboid. Click and hold the left mouse button to edit the label shape on the projection.

   

4. (Optional) Move one of the four points to change the size of the cuboid.

   

5. (Optional) To rotate the cuboid, select the middle point and then drag the cuboid up/down or to left/right.

   

Tracking with cuboids

To track with cuboids:

1. On the objects pane, select Draw new cuboid > select the label from the drop-down list > Track.

2. The cursor will be followed by a cuboid. Place the cuboid on the 3D scene.

3. Use projections to adjust the cuboid. Select and hold the left mouse button to edit the label shape on the projection.

   

4. (Optional) Move one of the four points to change the size of the cuboid.

   

5. (Optional) To rotate the cuboid, click on the middle point and then drag the cuboid up/down or to left/right.

   

6. Move several frames forward. You will see the cuboid you’ve added in frame 1. Adjust it, if needed.

7. Repeat to the last frame with the presence of the object you are tracking.

For more information about tracking, consult Track mode.

2.24 - Attribute annotation mode (basics)

• In this mode, you can edit attributes with fast navigation between objects and frames using a keyboard. Open the drop-down list in the top panel and select Attribute annotation.

  

• In this mode, objects panel change to a special panel:

  

• The active attribute will be red. In this case, it is gender. Look at the bottom side panel to see all possible shortcuts for changing the attribute. Press key 2 on your keyboard to assign a value (female) for the attribute or select from the drop-down list.

  

• Press Up Arrow/Down Arrow on your keyboard or select the buttons in the UI to go to the next/previous attribute. In this case, after pressing Down Arrow you will be able to edit the Age attribute.

  

• Use Right Arrow/Left Arrow keys to move to the previous/next image with annotation.

To display all the hot keys available in the attribute annotation mode, press F2. Learn more in the section attribute annotation mode (advanced).

2.25 - Filter

There are some reasons to use the feature:

1. When you use a filter, objects that don’t match the filter will be hidden.
2. The fast navigation between frames which have an object of interest. Use the Left Arrow / Right Arrow keys for this purpose or customize the UI buttons by right-clicking and select switching by filter. If there are no objects which correspond to the filter, you will go to the previous / next frame which contains any annotated objects.

To apply filters you need to click on the button on the top panel.

Create a filter

It will open a window for filter input. Here you will find two buttons: Add rule and Add group.

Rules

The Add rule button adds a rule for objects display. A rule may use the following properties:

Supported properties for annotation

• Supported properties for projects list
• Supported properties for tasks list
• Supported properties for jobs list
• Supported properties for cloud storages list

Supported operators for properties

== - Equally; != - Not equal; > - More; >= - More or equal; < - Less; <= - Less or equal;

Any in; Not in - these operators allow you to set multiple values in one rule;

Is empty; is not empty – these operators don’t require to input a value.

Between; Not between – these operators allow you to choose a range between two values.

Like - this operator indicate that the property must contain a value.

Starts with; Ends with - filter by beginning or end.

Some properties support two types of values that you can choose:

You can add multiple rules, to do so click the add rule button and set another rule. Once you’ve set a new rule, you’ll be able to choose which operator they will be connected by: And or Or.

All subsequent rules will be joined by the chosen operator. Click Submit to apply the filter or if you want multiple rules to be connected by different operators, use groups.

Groups

To add a group, click the Add group button. Inside the group you can create rules or groups.

If there is more than one rule in the group, they can be connected by And or Or operators. The rule group will work as well as a separate rule outside the group and will be joined by an operator outside the group. You can create groups within other groups, to do so you need to click the add group button within the group.

You can move rules and groups. To move the rule or group, drag it by the button. To remove the rule or group, click on the Delete button.

If you activate the Not button, objects that don’t match the group will be filtered out. Click Submit to apply the filter. The Cancel button undoes the filter. The Clear filter button removes the filter.

Once applied filter automatically appears in Recent used list. Maximum length of the list is 10.

Sort and filter lists

On the projects, task list on the project page, tasks, jobs, and cloud storage pages, you can use sorting and filters.

Sort by

You can sort by the following parameters:

• Jobs list: ID, assignee, updated date, stage, state, task ID, project ID, task name, project name.
• Tasks list or tasks list on project page: ID, owner, status, assignee, updated date, subset, mode, dimension, project ID, name, project name.
• Projects list: ID, assignee, owner, status, name, updated date.
• Cloud storages list: ID, provider type, updated date, display name, resource, credentials, owner, description.

To apply sorting, drag the parameter to the top area above the horizontal bar. The parameters below the horizontal line will not be applied. By moving the parameters you can change the priority, first of all sorting will occur according to the parameters that are above.

Pressing the Sort button switches Ascending sort/Descending sort.

Quick filters

Quick Filters contain several frequently used filters:

• Assigned to me - show only those projects, tasks or jobs that are assigned to you.
• Owned by me - show only those projects or tasks that are owned by you.
• Not completed - show only those projects, tasks or jobs that have a status other than completed.
• AWS storages - show only AWS cloud storages
• Azure storages - show only Azure cloud storages
• Google cloud storages - show only Google cloud storages

Date and time selection

When creating a Last updated rule, you can select the date and time by using the selection window.

You can select the year and month using the arrows or by clicking on the year and month. To select a day, click on it in the calendar, To select the time, you can select the hours and minutes using the scrolling list. Or you can select the current date and time by clicking the Now button. To apply, click Ok.

2.26 - Contextual images

Contextual images (or related images) are additional images that provide context or additional information related to the primary image.

Use them to add extra contextual about the object to improve the accuracy of annotation.

Contextual images are available for 2D and 3D tasks.

See:

• Folder structure
• Contextual images

Folder structure

To add contextual images to the task, you need to organize the images folder into one of the supported file layouts. A task with contextual images can be created both from an archive or from raw files.

Example for 2D tasks:

1. In the folder with the images for annotation, create a folder: related_images.
2. Add to the related_images a subfolder with the same name as the primary image to which it should be linked.
3. Place the contextual image(s) within the subfolder created in step 2.
4. Add folder to the archive.
5. Create task.

Supported file layouts for 2D and 3D tasks:

• 2D task
• 3D option 1
• 3D option 2
• 3D task KITTI format
• 3D task Supervisely format

root_directory/
  image_1_to_be_annotated.jpg
  image_2_to_be_annotated.jpg
  related_images/
    image_1_to_be_annotated_jpg/
      context_image_for_image_1.jpg
    image_2_to_be_annotated_jpg/
      context_image_for_image_2.jpg
    subdirectory_example/
      image_3_to_be_annotated.jpg
        related_images/
        image_3_to_be_annotated_jpg/
            context_image_for_image_3.jpg

root_directory/
  pointcloud1.pcd
  pointcloud1.jpg
  pointcloud2.pcd
  pointcloud2.png
  ...

root_directory/
  pointcloud1/
    pointcloud1.pcd
    pointcloud1_ri1.png
    pointcloud1_ri2.jpg
    ...
  pointcloud2/
    pointcloud2.pcd
    pointcloud2_ri1.bmp

image_00/
  data/
    0000000000.png
    0000000001.png
    0000000002.png
    0000000003.png
image_01/
  data/
    0000000000.png
    0000000001.png
    0000000002.png
    0000000003.png
image_N/
  data/
    0000000000.png
    0000000001.png
    0000000002.png
    0000000003.png
velodyne_points/
  data/
    0000000000.bin
    0000000001.bin
    0000000002.bin
    0000000003.bin

root_directory/
  pointcloud/
    pointcloud1.pcd
    pointcloud2.pcd
  related_images/
    pointcloud1_pcd/
      context_image_for_pointclud1.jpg
    pointcloud2_pcd/
      context_image_for_pointcloud2.jpg

For more general information about 3D data formats, see 3D data formats.

Contextual images

The maximum amount of contextual images is twelve.

By default they will be positioned on the right side of the main image.

When you add contextual images to the set, small toolbar will appear on the top of the screen, with the following elements:

Element	Description
	Fit views. Click to restore the layout to its original appearance. If you’ve expanded any images in the layout, they will returned to their original size. This won’t affect the number of context images on the screen.
	Add new image. Click to add context image to the layout.
	Reload layout. Click to reload layout to the default view. Note, that this action can change the number of context images resetting them back to three.

Each context image has the following elements:

2.27 - Shape grouping

This feature allows us to group several shapes.

You may use the Group Shapes button or shortcuts:

• G — start selection / end selection in group mode
• Esc — close group mode
• Shift+G — reset group for selected shapes

You may select shapes clicking on them or selecting an area.

Grouped shapes will have group_id filed in dumped annotation.

Also you may switch color distribution from an instance (default) to a group. You have to switch Color By Group checkbox for that.

Shapes that don’t have group_id, will be highlighted in white.

Shapes grouping video tutorial

2.28 - Dataset Manifest

Overview

When we create a new task in CVAT, we need to specify where to get the input data from. CVAT allows to use different data sources, including local file uploads, a mounted file share on the server, cloud storages and remote URLs. In some cases CVAT needs to have extra information about the input data. This information can be provided in Dataset manifest files. They are mainly used when working with cloud storages to reduce the amount of network traffic used and speed up the task creation process. However, they can also be used in other cases, which will be explained below.

A dataset manifest file is a text file in the JSONL format. These files can be created automatically with the special command-line tool, or manually, following the manifest file format specification.

How and when to use manifest files

Manifest files can be used in the following cases:

• A video file or a set of images is used as the data source and the caching mode is enabled. Read more
• The data is located in a cloud storage. Read more
• The predefined file sorting method is specified. Read more

The predefined sorting method

Independently of the file source being used, when the predefined sorting method is selected in the task configuration, the source files will be ordered according to the .jsonl manifest file, if it is found in the input list of files. If a manifest is not found, the order provided in the input file list is used.

For image archives (e.g. .zip), a manifest file (*.jsonl) is required when using the predefined file ordering. A manifest file must be provided next to the archive in the input list of files, it must not be inside the archive.

If there are multiple manifest files in the input file list, an error will be raised.

How to generate manifest files

CVAT provides a dedicated Python tool to generate manifest files. The source code can be found here.

Using the tool is the recommended way to create manifest files for you data. The data must be available locally to the tool to generate manifest.

Usage

usage: create.py [-h] [--force] [--output-dir .] source

positional arguments:
  source                Source paths

optional arguments:
  -h, --help            show this help message and exit
  --force               Use this flag to prepare the manifest file for video data
                        if by default the video does not meet the requirements
                        and a manifest file is not prepared
  --output-dir OUTPUT_DIR
                        Directory where the manifest file will be saved

Use the script from a Docker image

This is the recommended way to use the tool.

The script can be used from the cvat/server image:

docker run -it --rm -u "$(id -u)":"$(id -g)" \
  -v "${PWD}":"/local" \
  --entrypoint python3 \
  cvat/server \
  utils/dataset_manifest/create.py --output-dir /local /local/<path/to/sources>

Make sure to adapt the command to your file locations.

Use the script directly

Ubuntu 20.04

Install dependencies:

# General
sudo apt-get update && sudo apt-get --no-install-recommends install -y \
    python3-dev python3-pip python3-venv pkg-config

# Library components
sudo apt-get install --no-install-recommends -y \
    libavformat-dev libavcodec-dev libavdevice-dev \
    libavutil-dev libswscale-dev libswresample-dev libavfilter-dev

Create an environment and install the necessary python modules:

python3 -m venv .env
. .env/bin/activate
pip install -U pip
pip install -r utils/dataset_manifest/requirements.in

Examples

Create a dataset manifest in the current directory with video which contains enough keyframes:

python utils/dataset_manifest/create.py ~/Documents/video.mp4

Create a dataset manifest with video which does not contain enough keyframes:

python utils/dataset_manifest/create.py --force --output-dir ~/Documents ~/Documents/video.mp4

Create a dataset manifest with images:

python utils/dataset_manifest/create.py --output-dir ~/Documents ~/Documents/images/

Create a dataset manifest with pattern (may be used *, ?, []):

python utils/dataset_manifest/create.py --output-dir ~/Documents "/home/${USER}/Documents/**/image*.jpeg"

Create a dataset manifest using Docker image:

docker run -it --rm -u "$(id -u)":"$(id -g)" \
  -v ~/Documents/data/:${HOME}/manifest/:rw \
  --entrypoint '/usr/bin/bash' \
  cvat/server \
  utils/dataset_manifest/create.py --output-dir ~/manifest/ ~/manifest/images/

File format

The dataset manifest files are text files in JSONL format. These files have 2 sub-formats: for video and for images and 3d data.

Dataset manifest for video

The file describes a single video.

pts - time at which the frame should be shown to the user checksum - md5 hash sum for the specific image/frame decoded

{ "version": <string, version id> }
{ "type": "video" }
{ "properties": {
  "name": <string, filename>,
  "resolution": [<int, width>, <int, height>],
  "length": <int, frame count>
}}
{
  "number": <int, frame number>,
  "pts": <int, frame pts>,
  "checksum": <string, md5 frame hash>
} (repeatable)

Dataset manifest for images and other data types

The file describes an ordered set of images and 3d point clouds.

name - file basename and leading directories from the dataset root checksum - md5 hash sum for the specific image/frame decoded

{ "version": <string, version id> }
{ "type": "images" }
{
  "name": <string, image filename>,
  "extension": <string, . + file extension>,
  "width": <int, width>,
  "height": <int, height>,
  "meta": <dict, optional>,
  "checksum": <string, md5 hash, optional>
} (repeatable)

Example files

Manifest for a video

{"version":"1.0"}
{"type":"video"}
{"properties":{"name":"video.mp4","resolution":[1280,720],"length":778}}
{"number":0,"pts":0,"checksum":"17bb40d76887b56fe8213c6fded3d540"}
{"number":135,"pts":486000,"checksum":"9da9b4d42c1206d71bf17a7070a05847"}
{"number":270,"pts":972000,"checksum":"a1c3a61814f9b58b00a795fa18bb6d3e"}
{"number":405,"pts":1458000,"checksum":"18c0803b3cc1aa62ac75b112439d2b62"}
{"number":540,"pts":1944000,"checksum":"4551ecea0f80e95a6c32c32e70cac59e"}
{"number":675,"pts":2430000,"checksum":"0e72faf67e5218c70b506445ac91cdd7"}

Manifest for a dataset with images

{"version":"1.0"}
{"type":"images"}
{"name":"image1","extension":".jpg","width":720,"height":405,"meta":{"related_images":[]},"checksum":"548918ec4b56132a5cff1d4acabe9947"}
{"name":"image2","extension":".jpg","width":183,"height":275,"meta":{"related_images":[]},"checksum":"4b4eefd03cc6a45c1c068b98477fb639"}
{"name":"image3","extension":".jpg","width":301,"height":167,"meta":{"related_images":[]},"checksum":"0e454a6f4a13d56c82890c98be063663"}

2.29 - Data preparation on the fly

Description

Data on the fly processing is a way of working with data, the main idea of which is as follows: when creating a task, the minimum necessary meta information is collected. This meta information allows in the future to create necessary chunks when receiving a request from a client.

Generated chunks are stored in a cache of the limited size with a policy of evicting less popular items.

When a request is received from a client, the required chunk is searched for in the cache. If the chunk does not exist yet, it is created using prepared meta information and then put into the cache.

This method of working with data allows:

• reduce the task creation time.
• store data in a cache of the limited size with a policy of evicting less popular items.

Unfortunately, this method has several drawbacks:

• The first access to the data will take more time.
• It will not work for some videos, even if they have a valid manifest file. If there are not enough keyframes in the video for smooth video decoding, the task data chunks will be created with the default method, i.e. during the task creation.
• If the data has not been cached yet, and is not reachable during the access time, it cannot be retrieved.

How to use

To enable or disable this feature for a new task, use the Use Cache toggle in the task configuration.

Uploading a manifest with data

When creating a task, you can upload a manifest.jsonl file along with the video or dataset with images. You can see how to prepare it here.

2.30 - Shapes converter

The shapes converter is a feature that enables bulk actions on filtered shapes. It allows you to perform mutual conversion between masks, polygons and rectangles.

See:

• Run actions menu
• Convert shapes
• Convert shapes video tutorial

Run actions menu

Annotations actions can be accessed from the annotation menu. To access it, click on the burger icon and then select Run actions.

You will see the following dialog:

With the following fields:

Convert shapes

Recommended Precautions Before Running Annotation Actions

• Saving changes: It is recommended to save all changes prior to initiating the annotation action. If unsaved changes are detected, a prompt will advise to save these changes to avoid any potential loss of data.

• Disable auto-save: Prior to running the annotation action, disabling the auto-save feature is advisable. A notification will suggest this action if auto-save is currently active.

• Committing changes: Changes applied during the annotation session will not be committed to the server until the saving process is manually initiated. This can be done either by the user or through the auto-save feature, should it be enabled.

To convert shapes, do the following:

1. Annotate your dataset.

   

2. Set up filters.

   

3. From the burger menu, select Run actions.

4. Choose the action you need from the Select action drop-down list.

5. (Optional) In the Starting from frame field, enter the frame number where the action should begin, and in the up to frame field, specify the frame number where the action should end.

6. (Optional) Select an option from Or choose one of the predefined options to apply the action.

7. Click Run.
   A progress bar will appear. You may abort the process by clicking Cancel until the process commits modified objects at the end of pipeline.

   

Convert shapes video tutorial

3 - Automated Annotation

3.1 - Overview

Automatic annotation in CVAT is a tool that you can use to automatically pre-annotate your data with pre-trained models.

CVAT can use models from the following sources:

• Pre-installed models.
• Models integrated from Hugging Face and Roboflow.
• Self-hosted models deployed with Nuclio.
• AI agent functions (SAM2 tracking) for CVAT Online and Enterprise.

The following table describes the available options:

See:

• Running Automatic annotation
• Labels matching
• Models
• Adding models from Hugging Face and Roboflow

Running Automatic annotation

To start automatic annotation, do the following:

1. On the top menu, click Tasks.

2. Find the task you want to annotate and click Action > Automatic annotation.

   

3. In the Automatic annotation dialog, from the drop-down list, select a model.

4. Match the labels of the model and the task.

5. (Optional) In case you need the model to return masks as polygons, switch toggle Return masks as polygons.

6. (Optional) In case you need to remove all previous annotations, switch toggle Clean old annotations.

7. (Optional) You can specify a Threshold for the model. If not provided, the default value from the model settings will be used.

   

8. Click Annotate.

CVAT will show the progress of annotation on the progress bar.

You can stop the automatic annotation at any moment by clicking cancel.

Labels matching

Each model is trained on a dataset and supports only the dataset’s labels.

For example:

• DL model has the label car.
• Your task (or project) has the label vehicle.

To annotate, you need to match these two labels to give CVAT a hint that, in this case, car = vehicle.

If you have a label that is not on the list of DL labels, you will not be able to match them.

For this reason, supported DL models are suitable only for certain labels.

To check the list of labels for each model, see Models papers and official documentation.

Models

Automatic annotation uses pre-installed and added models.

List of pre-installed models:

Adding models from Hugging Face and Roboflow

In case you did not find the model you need, you can add a model of your choice from Hugging Face or Roboflow.

For more information, see Streamline annotation by integrating Hugging Face and Roboflow models.

This video demonstrates the process:

3.2 - Segment Anything 2 Tracker

Overview

Segment Anything 2 is a segmentation model that allows fast and precise selection of any object in videos or images. SAM2 tracking is available in two implementations:

1. Nuclio SAM2 Tracker: Available only for Enterprise deployments. This is implemented as a serverless function deployed via Nuclio framework.

2. AI Agent SAM2 Tracker: Available for CVAT Online and Enterprise via auto-annotation (AA) functions that run on user-side agents. This brings SAM2 tracking capabilities to CVAT Online users who previously couldn’t access this feature.

It is strongly recommended to deploy the model using a GPU. Although it is possible to use a CPU-based version, it generally performs much slower and is suitable only for handling a single parallel request. The AI agent variant runs on user hardware, providing flexibility for GPU usage without server configuration requirements.

Unlike a regular tracking model, both SAM2 tracker implementations are designed to be applied to existing objects (polygons and masks) to track them forward for a specified number of frames.

How to install

Choose the installation method based on your platform and deployment needs.

Nuclio SAM2 Tracker (CVAT Enterprise)

Docker

You can use existing scripts from the community repository (./serverless/deploy_cpu.sh or ./serverless/deploy_gpu.sh). To deploy the feature, simply run:

./serverless/deploy_gpu.sh "path/to/the/function"

Kubernetes

• You need to deploy the Nuclio function manually. Note that this function requires a Redis storage configured to keep the tracking state. You may use the same storage as cvat_redis_ondisk uses. When running the nuclio deploy command, make sure to provide the necessary arguments. The minimal command is:

nuctl deploy "path/to/the/function"
  --env CVAT_FUNCTIONS_REDIS_HOST="<redis_host>"
  --env CVAT_FUNCTIONS_REDIS_PORT="<redis_port>"
  --env CVAT_FUNCTIONS_REDIS_PASSWORD="<redis_password>" # if applicable

AI Agent SAM2 Tracker (CVAT Online + Enterprise)

The AI agent implementation enables SAM2 tracking for CVAT Online users and provides an alternative deployment method for Enterprise customers. This approach runs the tracking model on user hardware via auto-annotation (AA) functions.

Prerequisites

• Python 3.10 or later
• Git
• CVAT Online account or Enterprise instance
• Optional: NVIDIA GPU with CUDA support for faster inference

Setup Instructions

1. Clone the CVAT repository:

   git clone https://github.com/cvat-ai/cvat.git <CVAT_DIR>
   

2. Install required dependencies:

   pip install cvat-cli -r <CVAT_DIR>/ai-models/tracker/sam2/requirements.txt
   

   Note

   If you encounter issues installing SAM2, refer to the SAM2 installation guide for solutions to common problems.

3. Register the SAM2 function with CVAT:

   cvat-cli --server-host <CVAT_BASE_URL> --auth <USERNAME>:<PASSWORD> \
       function create-native "SAM2" \
       --function-file=<CVAT_DIR>/ai-models/tracker/sam2/func.py -p model_id=str:<MODEL_ID>
   

4. Run the AI agent:

   cvat-cli --server-host <CVAT_BASE_URL> --auth <USERNAME>:<PASSWORD> \
       function run-agent <FUNCTION_ID> \
       --function-file=<CVAT_DIR>/ai-models/tracker/sam2/func.py -p model_id=str:<MODEL_ID>
   

Parameter Reference

• <CVAT_BASE_URL>: Your CVAT instance URL (e.g., https://app.cvat.ai)
• <USERNAME> and <PASSWORD>: Your CVAT credentials
• <FUNCTION_ID>: The ID returned by the function create-native command
• <MODEL_ID>: A SAM2 model ID from Hugging Face Hub (e.g., facebook/sam2.1-hiera-tiny)

Optional Parameters

• GPU Support: Add -p device=str:cuda to the agent command to use NVIDIA GPU acceleration
• Organization Sharing: Add --org <ORG_SLUG> to both commands to share the function with your organization

Agent Behavior and Resilience

The AI agent runs as a persistent process on your hardware, providing several advantages:

• Hardware Independence: Runs outside the CVAT server, enabling tracking without server-side GPU/Nuclio installation
• Isolation: Agent crashes don’t affect the server; requests are retried or reassigned automatically
• Resource Control: You control the computational resources (CPU/GPU) used for tracking

Version Requirements

• AI Agent SAM2 Tracker: Requires CVAT version 2.42.0 or later
• Classic SAM2 Tracker: Available in all Enterprise versions
• Python: Version 3.10 or later for AI agent setup
• GPU Support: Optional but recommended for both implementations

Usage

Both SAM2 tracker implementations provide similar user experiences with slight differences in the UI labels.

Running the Nuclio SAM2 Tracker

The nuclio tracker can be applied to any polygons and masks. To run the tracker on an object, open the object menu and click Run annotation action.

Alternatively, you can use a hotkey: select the object and press Ctrl + E (default shortcut). When the modal opened, in “Select action” list, choose Segment Anything 2: Tracker:

Running the AI Agent SAM2 Tracker

Once you have registered the SAM2 AI agent and it’s running, you’ll see “AI Tracker: SAM2” as an available action in the annotation UI for video shape tracking.

To use the AI agent tracker:

1. Create or open a CVAT task from a video file or video-like sequence of images (all images must have the same dimensions)
2. Open one of the jobs from the task
3. Draw a mask or polygon around an object
4. Right-click the object and choose “Run annotation action”
5. Select “AI Tracker: SAM2” from the action list
6. Specify the target frame and click Run

The usage flow parallels the existing annotation action interface but utilizes the remote AI agent rather than built-in serverless functions.

Tracking Process

Specify the target frame until which you want the object to be tracked, then click the Run button to start tracking. The process begins and may take some time to complete. The duration depends on the inference device, and the number of frames where the object will be tracked.

Once the process is complete, the modal window closes. You can review how the object was tracked. If you notice that the tracked shape deteriorates at some point, you can adjust the object coordinates and run the tracker again from that frame.

Running on multiple objects

Instead of tracking each object individually, you can track multiple objects simultaneously. To do this, click the Menu button in the annotation view and select the Run Actions option:

Alternatively, you can use a hotkey: just press Ctrl + E (default shortcut) when there are no objects selected. This opens the actions modal. In this case, the tracker will be applied to all visible objects of suitable types (polygons and masks). In the action list of the opened modal, select either:

• Segment Anything 2: Tracker (for the nuclio implementation)
• AI Tracker: SAM2 (for the AI agent implementation)

Specify the target frame until which you want the objects to be tracked, then click the Run button to start tracking. The process begins and may take some time to complete. The duration depends on the inference device, the number of simultaneously tracked objects, and the number of frames where the objects will be tracked.

Once the process finishes, you may close the modal and review how the objects were tracked. If you notice that the tracked shapes deteriorate, you can adjust their coordinates and run the tracker again from that frame (for a single object or for many objects).

Limitations and Considerations

AI Agent Limitations

When using the AI agent implementation, keep in mind:

• Single Agent Constraint: Only one agent can run at a time for any given tracking function. Running multiple agents may cause random failures as they compete for tracking states.
• Memory-based State: Tracking states are kept in agent memory. If the agent crashes or is shut down, all tracking states are lost and active tracking processes will fail.
• Agent-only Usage: Tracking functions can only be used via agents. There is no equivalent of the cvat-cli task auto-annotate command for tracking.
• Rectangle Limitation: When using the AI Tools dialog (sidebar), only tracking functions that support rectangles will be selectable. The SAM2 tracker supports polygons and masks but not rectangles.
• Skeleton Tracking: Skeletons cannot currently be tracked by either implementation.

Tracker parameters

• Target frame: Objects will be tracked up to this frame. Must be greater than the current frame
• Convert polygon shapes to tracks: When enabled, all visible polygon shapes in the current frame will be converted to tracks before tracking begins. Use this option if you need tracks as the final output but started with shapes, produced for example by interactors (e.g. SAM2 or another one).

See Also

• SAM2 Object Tracking via AI Agent (Blog, July 31, 2025) - Detailed implementation and background information
• Auto-annotation Functions Documentation - Reference for creating custom tracking functions
• CVAT CLI Examples - Additional CLI usage examples