Automated Annotation

• 1: Overview
• 2: Segment Anything 2 Tracker
• 3: AI assistance
• 4: AI models

1 - Overview

Automatic annotation in CVAT is a tool that you can use to automatically pre-annotate your data with pre-trained models.

To use automatic annotation, you must be able to access at least one AI model of type “detector” or “reidentifier” on the CVAT instance. Consult AI models for information on available sources of AI models.

To start automatic annotation, do the following:

1. On the top menu, click Tasks.

2. Find the task you want to annotate and click Action > Automatic annotation.

   

3. In the Automatic annotation dialog, from the drop-down list, select a model.

4. Match the labels of the model and the task.

5. (Optional) In case you need the model to return masks as polygons, switch toggle Return masks as polygons.

6. (Optional) In case you need to remove all previous annotations, switch toggle Clean old annotations.

7. (Optional) You can specify a Threshold for the model. If not provided, the default value from the model settings will be used.

8. (Optional) For detector models in 2D tasks, specify a Region of interest. This limits annotation to a selected image area.

   

9. Click Annotate.

CVAT will show the progress of annotation on the progress bar.

You can stop the automatic annotation at any moment by clicking cancel.

Limiting automatic annotation input to a region of interest

For detector models in 2D tasks, you can restrict automatic annotation to a selected image area.

To set the region, enter Region of interest values: x, y, width, and height.

When a region of interest is set, CVAT sends only that image crop to the model. The resulting annotations are added back to the task in the correct full-frame coordinates.

Region of interest is supported for detector models, including Nuclio functions, Hugging Face and Roboflow models, and native AI-agent detector functions.

Region of interest is not supported for tracking or ReID functions. For task-level annotation, all processed frames must have the same resolution, and the selected region must fit inside every processed frame.

Labels matching

Each model is trained on a dataset and supports only the dataset’s labels.

For example:

• DL model has the label car.
• Your task (or project) has the label vehicle.

To annotate, you need to match these two labels to give CVAT a hint that, in this case, car = vehicle.

If you have a label that is not on the list of DL labels, you will not be able to match them.

For this reason, supported DL models are suitable only for certain labels.

To check the list of labels for each model, see Models papers and official documentation.

2 - Segment Anything 2 Tracker

Overview

Segment Anything 2 is a segmentation model that allows fast and precise selection of any object in videos or images. SAM2 tracking is available in two implementations:

1. Nuclio SAM2 Tracker: Available only for Enterprise deployments. This is implemented as a serverless function deployed via Nuclio framework.

2. AI Agent SAM2 Tracker: Available for CVAT Online and Enterprise via auto-annotation (AA) functions that run on user-side agents. This brings SAM2 tracking capabilities to CVAT Online users who previously couldn’t access this feature.

It is strongly recommended to deploy the model using a GPU. Although it is possible to use a CPU-based version, it generally performs much slower and is suitable only for handling a single parallel request. The AI agent variant runs on user hardware, providing flexibility for GPU usage without server configuration requirements.

Unlike a regular tracking model, both SAM2 tracker implementations are designed to be applied to existing objects (polygons and masks) to track them forward for a specified number of frames.

How to install

Choose the installation method based on your platform and deployment needs.

Nuclio SAM2 Tracker (CVAT Enterprise)

Docker

You can use existing scripts from the community repository (./serverless/deploy_cpu.sh or ./serverless/deploy_gpu.sh). To deploy the feature, simply run:

./serverless/deploy_gpu.sh "path/to/the/function"

Kubernetes

• You need to deploy the Nuclio function manually. Note that this function requires a Redis storage configured to keep the tracking state. You may use the same storage as cvat_redis_ondisk uses. When running the nuclio deploy command, make sure to provide the necessary arguments. The minimal command is:

nuctl deploy "path/to/the/function"
  --env CVAT_FUNCTIONS_REDIS_HOST="<redis_host>"
  --env CVAT_FUNCTIONS_REDIS_PORT="<redis_port>"
  --env CVAT_FUNCTIONS_REDIS_PASSWORD="<redis_password>" # if applicable

AI Agent SAM2 Tracker (CVAT Online + Enterprise)

The AI agent implementation enables SAM2 tracking for CVAT Online users and provides an alternative deployment method for Enterprise customers. This approach runs the tracking model on user hardware via auto-annotation (AA) functions. Deploy SAM2 using Docker Compose with pre-built images for a quick and straightforward setup.

Prerequisites

• Docker and Docker Compose
• CVAT Online account or Enterprise instance
• Optional: NVIDIA GPU with CUDA support for faster inference

Setup Instructions

The easiest way to deploy SAM2 with pre-built images from Docker Hub.

1. Clone the CVAT repository and navigate to the SAM2 agent directory:

   git clone https://github.com/cvat-ai/cvat.git
   cd cvat/ai-models/agents_deployment/sam2
   

2. Create or update the .env file with your configuration. For GPU deployment, set IMAGE_URL=cvat/sam2_agent:latest_GPU and COMPOSE_PROFILES=gpu. For CPU-only deployment, set IMAGE_URL=cvat/sam2_agent:latest and COMPOSE_PROFILES=cpu. Configure the remaining required variables (CVAT_BASE_URL, CVAT_ACCESS_TOKEN, FUNCTION_NAME, etc.) following the corresponding.

3. Start the agent:

   docker compose up
   

4. Verify the agent is running in the CVAT interface. You should see a new function model named <FUNCTION_NAME> in the list on the /models page and in the annotation actions list.

5. To stop and clean up:

   # Deregister the function from CVAT (must be called before volume removed)
   # Alternatively, you can always remove the function from CVAT interface
   docker compose run --rm cvat-function-deregister
   
   # Stop the agent and remove volumes
   docker compose down -v
   

For detailed configuration options and troubleshooting, see the Docker Compose agent guide.

Agent Behavior and Resilience

The AI agent runs as a persistent process on your hardware, providing several advantages:

• Hardware Independence: Runs outside the CVAT server, enabling tracking without server-side GPU/Nuclio installation
• Isolation: Agent crashes don’t affect the server; requests are retried or reassigned automatically
• Resource Control: You control the computational resources (CPU/GPU) used for tracking

Version Requirements

• AI Agent SAM2 Tracker: Requires CVAT version 2.42.0 or later
• Classic SAM2 Tracker: Available in all Enterprise versions
• GPU Support: Optional but recommended for both implementations

Usage

Both SAM2 tracker implementations provide similar user experiences with slight differences in the UI labels.

Running the Nuclio SAM2 Tracker

The nuclio tracker can be applied to any polygons and masks. To run the tracker on an object, open the object menu and click Run annotation action.

Alternatively, you can use a hotkey: select the object and press Ctrl + E (default shortcut). When the modal opened, in “Select action” list, choose Segment Anything 2: Tracker:

Running the AI Agent SAM2 Tracker

Once you have registered the SAM2 AI agent and it’s running, you’ll see “AI Tracker: SAM2” as an available action in the annotation UI for video shape tracking.

To use the AI agent tracker:

1. Create or open a CVAT task from a video file or video-like sequence of images (all images must have the same dimensions)
2. Open one of the jobs from the task
3. Draw a mask or polygon around an object
4. Right-click the object and choose “Run annotation action”
5. Select “AI Tracker: SAM2” from the action list
6. Specify the target frame and click Run

The usage flow parallels the existing annotation action interface but utilizes the remote AI agent rather than built-in serverless functions.

Tracking Process

Specify the target frame until which you want the object to be tracked, then click the Run button to start tracking. The process begins and may take some time to complete. The duration depends on the inference device, and the number of frames where the object will be tracked.

Once the process is complete, the modal window closes. You can review how the object was tracked. If you notice that the tracked shape deteriorates at some point, you can adjust the object coordinates and run the tracker again from that frame.

Running on multiple objects

Instead of tracking each object individually, you can track multiple objects simultaneously. To do this, click the Menu button in the annotation view and select the Run Actions option:

Alternatively, you can use a hotkey: just press Ctrl + E (default shortcut) when there are no objects selected. This opens the actions modal. In this case, the tracker will be applied to all visible objects of suitable types (polygons and masks). In the action list of the opened modal, select either:

• Segment Anything 2: Tracker (for the nuclio implementation)
• AI Tracker: SAM2 (for the AI agent implementation)

Specify the target frame until which you want the objects to be tracked, then click the Run button to start tracking. The process begins and may take some time to complete. The duration depends on the inference device, the number of simultaneously tracked objects, and the number of frames where the objects will be tracked.

Once the process finishes, you may close the modal and review how the objects were tracked. If you notice that the tracked shapes deteriorate, you can adjust their coordinates and run the tracker again from that frame (for a single object or for many objects).

Limitations and Considerations

AI Agent Limitations

When using the AI agent implementation, keep in mind:

• Single Agent Constraint: Only one agent can run at a time for any given tracking function. Running multiple agents may cause random failures as they compete for tracking states.
• Memory-based State: Tracking states are kept in agent memory. If the agent crashes or is shut down, all tracking states are lost and active tracking processes will fail.
• Agent-only Usage: Tracking functions can only be used via agents. There is no equivalent of the cvat-cli task auto-annotate command for tracking.
• Rectangle Limitation: When using the AI Tools dialog (sidebar), only tracking functions that support rectangles will be selectable. The SAM2 tracker supports polygons and masks but not rectangles.
• Skeleton Tracking: Skeletons cannot currently be tracked by either implementation.

Tracker parameters

• Target frame: Objects will be tracked up to this frame. Must be greater than the current frame
• Convert polygon shapes to tracks: When enabled, all visible polygon shapes in the current frame will be converted to tracks before tracking begins. Use this option if you need tracks as the final output but started with shapes, produced for example by interactors (e.g. SAM2 or another one).

See Also

• SAM2 Object Tracking via AI Agent (Blog, July 31, 2025) - Detailed implementation and background information
• Auto-annotation Functions Documentation - Reference for creating custom tracking functions
• CVAT CLI Examples - Additional CLI usage examples

3 - AI assistance

Use AI assistance to annotate images and videos with visual prompts, automatic models, and tracking.

The AI assistance dialog has three tabs: Prompting, Auto-labeling, and Tracking. In each tab, the Model drop-down includes Add your own model, which opens the AI agent setup instructions in a new browser tab.

See:

• Prompting
  • Annotate with visual prompts
  • Limit visual prompts to a region of interest
  • Add extra points
  • Delete points
  • Settings
• Auto-labeling
  • Labels matching
  • Annotate automatically
  • Limit automatic annotation to a region of interest
• Tracking
  • Track objects
  • When tracking

Prompting

Use Prompting to create masks or polygons from points or bounding boxes, depending on the model. These models are called interactors in the SDK.

For models that support point prompts:

• Positive points define the area in which the object is located.
• Negative points define the area in which the object is not located.

Annotate with visual prompts

To annotate with visual prompts, do the following:

1. Click Magic wand , and go to the Prompting tab.

2. From the Model drop-down, select a model (see AI models).
   Click the Question mark to see information about each model:

   

3. From the Label drop-down, select a label for the object.

4. (Optional) If the model returns masks, and you need to convert masks to polygons, use the Convert masks to polygons toggle.

5. Click Add visual prompts.

6. Use the left click to add positive points and the right click to add negative points.
   Number of points you can add depends on the model.

7. On the top menu, click Done (or Shift+N, N).

Limit visual prompts to a region of interest

For image/video jobs, you can restrict visual prompts to a selected image area. Use this when you want the model to segment only a specific part of the frame.

To set the region:

1. In the Prompting tab, specify Region of interest values: x, y, width, and height.

2. Alternatively, click Draw a region of interest and draw the area on the canvas.

   

3. Click Add visual prompts and place points or boxes inside the selected region.

When a region of interest is set, CVAT sends only that image crop to the model, restricts visual prompts to the selected area, and adds the resulting shape back in the correct full-frame position.

Add extra points

Each model has a minimum required number of points for annotation. Once the required number of points is reached, the request is automatically sent to the server. The server processes the request and adds a polygon to the frame.

For a more accurate outline, postpone request to finish adding extra points first:

1. Hold down the Ctrl key.
   On the top panel, the Block button will turn blue.
2. Add points to the image.
3. Release the Ctrl key, when ready.

In case you used Mask to polygon when the object is finished, you can edit it like a polygon.

You can change the number of points in the polygon with the slider:

Delete points

To delete a point, do the following:

1. With the cursor, hover over the point you want to delete.
2. If the point can be deleted, it will enlarge and the cursor will turn into a cross.
3. Left-click on the point.

Settings

• On how to adjust the polygon, see Objects sidebar.

• For more information about polygons in general, see Annotation with polygons.

Auto-labeling

Use Auto-labeling to annotate a frame automatically. Depending on the model, results can include bounding boxes, masks, tags, or poses.

Labels matching

Each model is trained on a dataset and supports only the dataset’s labels.

For example:

• DL model has the label car.
• Your task (or project) has the label vehicle.

To annotate, you need to match these two labels to give DL model a hint, that in this case car = vehicle.

If you have a label that is not on the list of DL labels, you will not be able to match them.

You can check each deployed model’s supported labels at the Models page.

Annotate automatically

To annotate automatically, do the following:

1. Click Magic wand , and go to the Auto-labeling tab.

2. From the Model drop-down, select model (see AI models).

3. From the left drop-down select the DL model label, from the right drop-down select the matching label of your task.

   

4. (Optional) If the model returns masks, and you need to convert masks to polygons, use the Convert masks to polygons toggle.

5. (Optional) You can specify a Threshold for the model. If not provided, the default value from the model settings will be used.

6. Click Annotate.

This action will automatically annotate one frame. For automatic annotation of multiple frames, see Automatic annotation.

Limit automatic annotation to a region of interest

For image/video jobs, you can restrict automatic annotation to a selected image area. Use this when only part of the frame should be analyzed.

To set the region:

1. In the Auto-labeling tab, specify Region of interest values: x, y, width, and height.

2. Alternatively, click Draw a region of interest and draw the area on the canvas.

   

3. Click Annotate.

CVAT sends only the selected image crop to the model and maps the returned annotations back to the correct full-frame coordinates.

Tracking

Use Tracking to follow objects across frames in a video or image sequence.

Track objects

To track objects, do the following:

1. Click Magic wand , and go to the Tracking tab.

   

2. From the Model drop-down, select a tracker.

3. From the Label drop-down, select the label for the object.

4. Click Track, and annotate the objects with the bounding box in the first frame.

5. Go to the top menu and click Next (or the F on the keyboard) to move to the next frame.
   All annotated objects will be automatically tracked.

When tracking

• To enable/disable tracking, use Tracker switcher on the sidebar.

  

• Trackable objects have an indication on canvas with a model name.

  

• You can follow the tracking by the messages appearing at the top.

  

4 - AI models

CVAT’s AI features use AI models, which can be added via several methods, namely:

• Nuclio functions.
• Functions powered by third-party services (Hugging Face and Roboflow).
• Functions powered by CVAT AI agents (also known as native functions).

The following table shows the availability of each model deployment method across CVAT editions:

Each function has a certain kind, which determines the tools that can use it in the CVAT UI. The following kinds are defined:

Nuclio functions

A Nuclio function is a Docker container that runs on the same host or Kubernetes cluster as CVAT itself and processes inference requests that come from it. To add a Nuclio function to CVAT, a system administrator must build and deploy it. Consult the serverless tutorial for more information.

Once deployed, a Nuclio function is available to every user of the CVAT instance.

Nuclio functions can have any kind.

A Nuclio function can be built from custom source code and thus be internally implemented in any way that fits the interface expected by CVAT. However, the CVAT source repository also includes several predefined Nuclio functions in source code form, which can be used either directly or as examples for building custom functions. In addition, CVAT Enterprise customers gain access to additional predefined functions, including the SAM 2 Tracker. Consult Segment Anything 2 Tracker for more information.

Predefined Nuclio functions

The following predefined function implementations are available in the serverless directory of the source repository.

Interactors

Model	Description	Example
Inside-Outside Guidance (IOG)	The model uses a bounding box and inside/outside points to create a mask. First of all, you need to create a bounding box, wrapping the object. Then you need to use positive and negative points to say the model where is a foreground, and where is a background. Negative points are optional. For more information, see: GitHub: IOG Paper: IOG
Segment Anything Model (SAM)	The Segment Anything Model (SAM) produces high quality object masks, and it can be used to generate masks for all objects in an image. It has been trained on a dataset of 11 million images and 1.1 billion masks, and has strong zero-shot performance on a variety of segmentation tasks. For more information, see: GitHub: Segment Anything Site: Segment Anything Paper: Segment Anything

Detectors

Trackers

Model	Description	Example
Transformer Tracking (TransT)	Simple and efficient online tool for object tracking and segmentation. If the previous frame was the latest keyframe for the object, the trackable object will be tracked automatically. This is a modified version of the PyTracking Python framework based on Pytorch For more information, see: GitHub: TransT Paper: TransT

Nuclio functions in CVAT Online

Since CVAT Online is administered by CVAT.ai Corporation, users may not add custom Nuclio functions to it. However, CVAT.ai makes several functions from the lists above available in CVAT Online. The exact set of available functions may vary over time at CVAT.ai’s discretion.

Third-party functions

A third-party function is a resource stored in CVAT that comprises a URL to a model in a third-party inference service and a credential used to access that service. This third-party service is called when the function is used as a model in the CVAT UI.

The following services are currently supported:

• Hugging Face.

  To be usable from CVAT, a model hosted in Hugging Face must support the Inference API, which can be verified by checking that the “Inference Providers” gadget on the model page lists “HF Inference API” as one of the providers. Additionally, the model’s task must be “Image Classification”, “Image Segmentation”, or “Object Detection”.

• Roboflow.

  To be usable from CVAT, a model hosted in Roboflow must have task type “Classification”, “Instance Segmentation”, “Object Detection”, or “Semantic Segmentation”.

A third-party function can be created via the CVAT UI on the Models page.

Any user may create a third-party function to their personal workspace, in which case it will only be usable by them.

Alternatively, an organization maintainer may create a third-party function within the organization. In this case, the function will be usable by all other members of the organization.

Third-party functions always have kind “detector”.

For a walkthrough of this feature, see Streamline annotation by integrating Hugging Face and Roboflow models.

This video demonstrates the process:

Native functions

You can connect your own model to CVAT using cvat-cli. The model runs on your machine as an AI agent: it receives requests from CVAT and sends back annotations. CVAT stores the model’s name, labels, and capabilities as a native function.

Agents support three kinds of functions:

• Detectors run automatically and produce boxes, masks, tags, or poses, depending on the model.
• Interactors use your prompts, such as points and bounding boxes, to annotate objects in AI assistance → Prompting.
• Trackers follow objects across frames, through Tracking or annotation actions depending on their shape types.

The setup has three steps:

1. Prepare a Python function for your model, or choose a ready-to-use function below.
2. Choose a deployment scope and register the model with cvat-cli function create-native.
3. Start the agent with cvat-cli function run-agent and keep it running while using the model in CVAT.

Set up CVAT CLI

Run these commands on the machine that will run the model. The examples use a Linux/macOS shell and require Git and a Python version supported by the model’s dependencies.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip cvat-cli

Create a Personal Access Token in CVAT, then configure your connection:

export CVAT_SERVER_URL="https://app.cvat.ai"
export CVAT_ACCESS_TOKEN="replace-with-your-token"

For Enterprise, use your instance URL instead of https://app.cvat.ai.

Choose a deployment scope

Choose one of these settings before running the examples.

• Personal workspace
• Organization workspace
• Global deployment

export CVAT_ORG=""

export CVAT_ORG="your-organization-slug"

export CVAT_ORG=""

Ready-to-use models

The cvat-ai/cvat-models repository provides Python functions for supported models. You can use them directly without writing a wrapper:

Clone the repository and run the examples from its directory:

git clone https://github.com/cvat-ai/cvat-models.git
cd cvat-models

Choose an example below. After registration, replace <function_id> (including the angle brackets) with the numeric ID shown in the command output. For example, if the output says Created function #8, use 8. Use the same model parameters and workspace when registering and starting the agent. The examples use the CPU; for a GPU, install compatible CUDA-enabled PyTorch dependencies and change -p device=str:cpu to -p device=str:cuda in both commands.

• Ultralytics YOLO
• Hugging Face Transformers
• SAM2 interactor
• SAM2 tracking

After setup

Keep the agent terminal open while using the model. Ctrl+C stops the agent but leaves the model registered; run the same run-agent command to restart it. To remove the model after stopping the agent:

cvat-cli --server-host "$CVAT_SERVER_URL" --org "$CVAT_ORG" function delete <function_id>

If the model is missing, check the selected server and workspace. If requests wait without results, check the agent’s terminal and function ID. In older CVAT versions, AI assistance → Auto-labeling is named AI Tools → Detectors.

Use another model

For a model without a ready-to-use function, implement the CVAT SDK function interface. Start with the Python detection function example, which shows how to load a model, define its labels, and return predictions in CVAT format. For prompt-based models or trackers, see the interaction protocol or tracking protocol.

The CLI function-loading examples explain how to expose a function through a Python file or a create factory and pass parameters with -p. Then use the same registration and agent commands with your Python file and its parameters. Native functions can be detectors, interactors, or trackers.

For container deployment, see the Docker Compose guide or Helm guide.