This is the multi-page printable view of this section. Click here to print.
Advanced
- 1: CVAT Architecture
- 2: CVAT deployment on Kubernetes with Helm
- 3: Semi-automatic and Automatic Annotation
- 4: CVAT Analytics and monitoring
- 5: Mounting cloud storage
- 6: LDAP Backed Authentication
- 7: Backup guide
- 8: Upgrade guide
- 9: Webhooks
- 10: Custom Certificates
1 - CVAT Architecture
This guide is designed to provide a comprehensive overview of the architecture and components of the CVAT and to illustrate how each component interacts within the system.
Domain | Component | Functionality | Description |
---|---|---|---|
Analytics | Vector | Event processing | There are several components that process events (backend, frontend, web UI). All events are sent to a single point - Vector, where they are processed and then redirected to ClickHouse. For more information, see Analytics. |
ClickHouse | Event database | Stores events. For more information, see Analytics. | |
Grafana | Dashboards | Data based on the web interface. For more information, see Analytics. | |
Data storage NFS | RVVX access mode storage is required in case of multi-node deployment. Available with different types of storages: |
Contains data required for CVAT operations | It is necessary to have the capability for multiple mounting (across several nodes) in RWX mode. For more information, see K8 Deployment with Helm |
Data cache | Apache kvrocks | Used for data caching (queries and search). Suitable for environments that require frequent database queries. | Apache Kvrocks |
Job queue | Redis | Queue manager | |
Database | PostgreSQL | Database | A database where data is stored in a structured form. |
CVAT.ai Components | Ingress Controller (can be disabled) | Routing traffic. | CVAT deployment on Kubernetes with Helm |
Authorization | Authorization service based on Open Policy Agent. | ||
Backend CVAT | Backend | Main framework | Main engine, uses Django + Django DRF. |
Workers | Import Worker | Everything related to loading data - creating tasks, uploading annotations, etc. | |
Export Worker | Everything related to exporting data - exporting results, creating dumps, etc. | ||
Annotation Worker | Auto-annotation tasks. | ||
Utils Worker | Responsible for tracking various file changes and more. | ||
Analytics Report | Reports and analytics that are displayed in the CVAT interface. | ||
Quality Report | Analysis and reports on data quality. | ||
Webhook Worker | Manages webhooks. | ||
Auto annotation | Auto Annotation Nucio | Microservice application, used for auto annotation. | How to enable auto annotation feature. |
2 - CVAT deployment on Kubernetes with Helm
- Prerequisites
- Configuration
- Deployment
- Post-deployment configuration
- FAQ
- What is kubernetes and how it is working?
- What is helm and how it is working?
- How to setup Minikube
- How to understand what diff will be inflicted by ‘helm upgrade’?
- I want to use my own postgresql/redis with your chart.
- I want to override some settings in values.yaml.
- Why you used external charts to provide redis and postgres?
Prerequisites
- Installed and configured kubernetes cluster. If you do not already have a cluster, you can create one by using Minikube. How to setup Minikube.
- Installed kubectl
- Installed Helm.
- Installed dependencies
Installing dependencies
To install and/or update run:
Optional steps
-
Ingress configuration for the Traefik ingress controller is enabled by default.
Note for Minikube use:
- because the Traefik creates its main service with
Loadbalanser
type, which involve the assignment of externalIP by Cloud, what never happens on Minikube, you need to explicitly set the externalIP address for the traefic service. Add the following tovalues.override.yaml
file: - Also ensure that your CVAT ingress appears on your hosts file (/etc/hosts).
You can do this by running this command:
cvat.local
is default domainname, you can override it viavalues.override.yaml
.
- because the Traefik creates its main service with
Configuration
- Create
values.override.yaml
file insidehelm-chart
directory. - Fill
values.override.yaml
with new parameters for chart. - Override postgresql password
Postgresql password?
Put below into your values.override.yaml
Or create your own secret and use it with:
(Optional) Enable Auto annotation feature
Before starting, ensure that the following prerequisites are met:
- The Nuclio CLI (nuctl) is installed. To install the CLI, simply download the appropriate CLI version to your installation machine.
-
Set
nuclio.enabled: true
in yourvalues.override.yaml
-
Run
helm dependency update
inhelm-chart
directory -
Because Nuclio functions are images that need to be pushed and pulled to/from the registry, you need to configure credentials to pull from your preferable registry with the following settings: Options:
-
values.override.yaml
file: -
Or you can create a secret with credentials as described in the guide and set
registry.secretName=your-registry-credentials-secret-name
in thevalues.override.yaml
file. -
In the case of using Minikube, you can run a local unsecured registry with minikube add-ons:
Before Docker container images can be pushed to your newly created insecure registry, you need to add its address (
$(minikube ip):5000
) to the list of insecure registries to instruct Docker to accept working against it: follow the instructions in the Docker documentation
You might also need to log into your registry account (docker login) on the installation machine before running the deployment command.
-
-
Create cvat project:
-
Finally deploy the function, i.e.:
- using minikube registry:
- using Docker hub:
- using minikube registry:
Analytics
Analytics is enabled by default, to disable set analytics.enabled: false
in your values.override.yaml
Deployment
Make sure you are using correct kubernetes context. You can check it with kubectl config current-context
.
Warning: The k8s service name of Open Policy Agent is fixed to opa by default. This is done to be compatible with CVAT 2.0 but limits this helm chart to a single release per namespace. The OPA url currently can´t be set as an environment variable. As soon as this is possible you can set cvat.opa.composeCompatibleServiceName to false in your value.override.yaml and configure the opa url as additional env.
Execute following command from repo root directory
With overrides:
helm upgrade -n <desired_namespace> <release_name> -i --create-namespace ./helm-chart -f ./helm-chart/values.yaml -f ./helm-chart/values.override.yaml
Without overrides:
helm upgrade -n <desired_namespace> <release_name> -i --create-namespace ./helm-chart -f ./helm-chart/values.yaml
Post-deployment configuration
- Create super user
How to create superuser?
FAQ
What is kubernetes and how it is working?
What is helm and how it is working?
See https://helm.sh/
How to setup Minikube
- Please follow the official Minikube installation guide
-
How to understand what diff will be inflicted by ‘helm upgrade’?
You can use https://github.com/databus23/helm-diff#install for that
I want to use my own postgresql with your chart.
Just set postgresql.enabled
to false
in the override file, then put the parameters of your database
instance in the external
field.
You may also need to configure username
, database
and password
fields
to connect to your own database:
In example above corresponding secret will be created automatically, but if you want to use existing secret change secret.create
to false
and set name
of existing secret:
The secret must contain the database
, username
and password
keys to access to the database
like:
I want to use my own redis with your chart.
Just set redis.enabled
to false
in the override file, then put the parameters of your Redis
instance in the external
field.
You may also need to configure password
field to connect to your own Redis:
In the above example the corresponding secret will be created automatically, but if you want to use an existing secret
change secret.create
to false
and set name
of the existing secret:
The secret must contain the redis-password
key like:
I want to override some settings in values.yaml.
Just create file values.override.yaml
and place your changes here, using same structure as in values.yaml
.
Then reference it in helm update/install command using -f
flag
Why you used external charts to provide redis and postgres?
Because they definitely know what they do better then we are, so we are getting more quality and less support
How to use custom domain name with k8s deployment:
The default value cvat.local
may be overridden with --set ingress.hosts[0].host
option like this:
How to fix fail of helm upgrade
due label field is immutable reason?
If an error message like this:
To fix that, delete CVAT Deployments before upgrading
How to use existing PersistentVolume to store CVAT data instead of default storage
It is assumed that you have created a PersistentVolumeClaim named my-claim-name
and a PersistentVolume that backing the claim.
Claims must exist in the same namespace as the Pod using the claim.
For details see.
Add these values in the values.override.yaml
:
3 - Semi-automatic and Automatic Annotation
⚠ WARNING: Do not use
docker compose up
If you did, make sure all containers are stopped bydocker compose down
.
-
To bring up cvat with auto annotation tool, from cvat root directory, you need to run:
If you did any changes to the Docker Compose files, make sure to add
--build
at the end.To stop the containers, simply run:
-
You have to install
nuctl
command line tool to build and deploy serverless functions. Download version 1.13.0. It is important that the version you download matches the version in docker-compose.serverless.yml. For example, using wget.After downloading the nuclio, give it a proper permission and do a softlink.
-
Deploy a couple of functions. This will automatically create a
cvat
Nuclio project to contain the functions. Commands below should be run only after CVAT has been installed usingdocker compose
because it runs nuclio dashboard which manages all serverless functions.GPU Support
You will need to install Nvidia Container Toolkit. Also you will need to add
--resource-limit nvidia.com/gpu=1 --triggers '{"myHttpTrigger": {"maxWorkers": 1}}'
to the nuclio deployment command. You can increase the maxWorker if you have enough GPU memory. As an example, below will run on the GPU:Note:
- The number of GPU deployed functions will be limited to your GPU memory.
- See deploy_gpu.sh script for more examples.
- For some models (namely SiamMask) you need an Nvidia driver version greater than or equal to 450.80.02.
Note for Windows users:
If you want to use nuclio under Windows CVAT installation you should install Nvidia drivers for WSL according to this instruction and follow the steps up to “2.3 Installing Nvidia drivers”. Important requirement: you should have the latest versions of Docker Desktop, Nvidia drivers for WSL, and the latest updates from the Windows Insider Preview Dev channel.
Troubleshooting Nuclio Functions:
-
You can open nuclio dashboard at localhost:8070. Make sure status of your functions are up and running without any error.
-
Test your deployed DL model as a serverless function. The command below should work on Linux and Mac OS.
-
To check for internal server errors, run
docker ps -a
to see the list of containers. Find the container that you are interested, e.g.,nuclio-nuclio-tf-faster-rcnn-inception-v2-coco-gpu
. Then check its logs bydocker logs <name of your container>
e.g., -
To debug a code inside a container, you can use vscode to attach to a container instructions. To apply your changes, make sure to restart the container.
4 - CVAT Analytics and monitoring
CVAT Analytics suite of tools is designed to track and understand users’ behavior, system performance, and for identifying potential issues in your application.
You can also visualize user activity through Grafana, and aggregate user working time by the jobs.
Gathered logs can be additionally filtered for efficient debugging.
By using analytics, you’ll gain valuable insights to optimize your system and enhance user satisfaction.
CVAT analytics are available from the top menu.
Superusers and users with administrator role have access to analytics.
Permission to access analytics can also be granted when editing a user
on admin page by Has access to analytics
checkbox.
Note: CVAT analytics and monitoring are available only for on-prem solution.
See:
High-level architecture
The CVAT analytics is based on Vector, ClickHouse, and Grafana.
CVAT Analytics
CVAT and its analytics module can be set up locally, for self-hosted solution analytics are enabled by default.
For detailed CVAT installation instructions, see Installation Guide or refer to the CVAT Course for installation videos.
All analytics-related features will be launched when you start CVAT containers with the following command:
Ports settings
If you cannot access analytics on development environment, see Analytics Ports
Events log structure
Relational database schema with the following fields:
Field | Description |
---|---|
scope | Scope of the event (e.g., zoomin:image , add:annotations , delete:image , update:assignee ). |
obj_name | Object name or None (e.g., task, job, cloudstorage, model, organization). |
obj_id | Object identifier as in DB or None. |
obj_val | Value for the event as string or None (e.g., frame number, number of added annotations). |
source | Who generates the log event (e.g., server, ui). |
timestamp | Local event time (in general for UI and server, the time is different). |
count | How many times in the row it occurs. |
duration | How much time does it take (it can be 0 for events without duration). |
project_id | Project ID or None. |
task_id | Task ID or None. |
job_id | Job ID or None. |
user_id | User ID or None. |
user_name | User name or None. |
user_email | User email or None. |
org_id | Organization ID or None. |
org_slug | Organization slug or None. |
payload | JSON payload or None. Extra fields can be added to the JSON blob. |
Types of supported events
Supported events change the scope of information displayed in Grafana.
Server events:
-
create:project
,update:project
,delete:project
-
create:task
,update:task
,delete:task
-
create:job
,update:job
,delete:job
-
create:organization
,update:organization
,delete:organization
-
create:user
,update:user
,delete:user
-
create:cloudstorage
,update:cloudstorage
,delete:cloudstorage
-
create:issue
,update:issue
,delete:issue
-
create:comment
,update:comment
,delete:comment
-
create:annotations
,update:annotations
,delete:annotations
-
create:label
,update:label
,delete:label
-
export:dataset
,import:dataset
-
call:function
-
create:membership
,update:membership
,delete:membership
-
create:webhook
,update:webhook
,delete:webhook
-
create:invitation
,delete:invitation
Client events:
-
load:cvat
-
load:job
,save:job
-
send:exception
-
draw:object
,paste:object
,copy:object
,propagate:object
,drag:object
,resize:object
,delete:object
,merge:objects
,split:objects
,group:objects
,slice:object
,join:objects
-
change:frame
-
zoom:image
,fit:image
,rotate:image
-
action:undo
,action:redo
-
run:annotations_action
-
click:element
-
debug:info
Working time calculation
Here is a short overview of how CVAT deals with the user’s working time:
-
The user interface collects events when a user interacts with the interface (resizing canvas, drawing objects, clicking buttons, etc) The structure of one single event is described here.
-
The user interface sends these events in bulk to the server. Currently, it uses the following triggers to send events:
- Periodical timer (~90 seconds)
- A user clicks the “Save” button on the annotation view
- A user opens the annotation view
- A user closes the annotation view (but not the tab/browser)
- A user clicks Logout button
-
When events reach the server, it calculates working time based on timestamps of the events.
-
The working time for an event is computed as the sum of the following:
- The difference between the start time of the event and the end time of the previous event, if it is not more than 100 seconds.
- The duration of the event, for events of type
change:frame
.
-
After calculation, the server generates
send:working_time
events with time value in payload. These events may or may not be bound to a certain job/task/project, depending on the client-side events that were used to generate them. -
CVAT saves the event in the database and later these events are used to compute metrics for analytics.
Request id
for tracking
Note, that every response to an API request made to the
the server includes a header named X-Request-Id
,
for example: X-Request-Id: 6a2b7102-c4b9-4d57-8754-5658132ba37d
.
This identifier is also recorded in all server events that occur as a result of the respective request.
For example, when an operation to create a task is performed, other related entities such as labels and attributes are generated on the server in addition to the Task object.
All events associated with this operation will have the same request_id
in
the payload field.
Fetching event data as CSV from the /api/events
endpoint
The /api/events
endpoint allows the fetching of
event data with filtering parameters such as
org_id
, project_id
, task_id
, job_id
, and user_id
.
For more details, see Swagger API Documentation.
For example, to fetch all events associated with a specific job,
the following curl
command can be used:
In the response, you will receive a query ID:
As this process may take some time to complete,
the status of the request can be checked by
adding the query parameter query_id
to the request:
Upon successful creation, the server will return a 201 Created
status:
HTTP/2 201
allow: GET, POST, HEAD, OPTIONS
date: Tue, 16 May 2023 13:38:42 GMT
referrer-policy: same-origin
server: Apache
vary: Accept,Origin,Cookie
x-content-type-options: nosniff
x-frame-options: DENY
x-request-id: 4631f5fa-a4f0-42a8-b77b-7426fc298a85
The CSV file can be downloaded by
adding the action=download
query parameter to the request:
This will download and save the file to /tmp/events.csv
on your local machine.
Dashboards
By default, three dashboards are available in CVAT.
To access them, click General, you will be forwarded to the page with available dashboards.
Dashboard | Description |
---|---|
All Events | Dashboard that shows all event logs, timestamps, and source. |
Management | Dashboard with information about user activities such as working time by job and so on. |
Monitoring | Dashboard showing server logs, including errors. |
Dashboard: All Events
The dashboard shows all events, their timestamps, and their source.
Element | Description |
---|---|
Filters | Can be used as drop-down lists or search fields. Click on the arrow to activate. |
Overall activity | Graph that shows the overall activity by the selected filters. |
Scope | Users’ activity, see Types of supported events. |
obj_name | Object or item related to the Scope. |
obj_id | Object’s id. Might be empty. |
source | Source of the event, can be client or server . |
timestamp | Time when the event happened. |
count | Common field for all events, not null where it makes sense, for example, the number of saved objects in an annotation. |
duration | Duration in milliseconds. |
project_id | Id of the project. |
project_id | Id of the project. |
task_id | ID of the task. |
job_id | ID of the job. |
There are two fields with statistics at the bottom of the dashboard, about browser and OS users use.
Click on the column name to enable a filter.
If you want to inspect the value, hover over it and click on the eye icon.
Dashboard: Management
The dashboard shows user activity.
Element | Description |
---|---|
Filters | Can be used as drop-down lists or search fields. Click on the arrow to activate. |
User activity | Graph that shows when the user was active (data and time), click on the user id below, to see the graph for the dedicated user. |
Overall activity | Graph shows common activity for all users. |
User | User ID. |
Project | Project ID. Might be empty. |
Task | Task ID. Might be empty. |
Job | Job ID. Might be empty. |
Working time(h) | Time spent on task in hours. |
Activity | Number of events for each user. |
Click on the column name to enable a filter.
If you want to inspect the value, hover over it and click on the eye icon.
Dashboard: Monitoring
The dashboard shows server logs, helps handle errors, and shows user activity.
Element | Description |
---|---|
Filters | Can be used as drop-down lists or search fields. Click on the arrow to activate. |
Active users (now) | Number of active users on an instance. |
Overall activity | Graph that shows the number of active users. |
Exceptions | Graph that shows the number of errors that happened in the instance. |
timestamp | Time when the error happened. |
user_id | User ID. |
user_name | User nickname. |
project_id | Id of the project. Might be empty. |
task_id | Task ID. Might be empty. |
job_id | Job ID. Might be empty. |
error | Error description |
stack | Error description |
payload | Error description |
stack | Stack trace, which is a report of the active stack frames at a certain point in time during the execution. This information is typically used for debugging purposes to locate where an issue occurred. |
payload | JSON that describes the entire object, which contains several properties. This data in the payload is related to an event that was created as a result of a failed API request. The payload contains information about this event. |
Click on the column name to enable a filter.
If you want to inspect the value, hover over it and click on the eye icon.
Dashboards setup
You can adjust the dashboards. To do this, click on the graph or table name and from the drop-down menu select Edit.
Adjust the query in the editor.
Example of query:
Note, that by default the updated configuration will not be saved and will be reset to the default parameters after you restart the container.
To save the updated configuration, do the following:
-
Update Configuration: Start by making your desired changes in the query.
-
Apply Changes: Once you’ve made your changes, click the Apply button to ensure the changes are implemented.
-
Save Configuration: To save your applied changes, on the top of the dashboard, click the Save button.
-
Replace Configuration File: After saving, replace the existing Grafana dashboard configuration file is located at
components/analytics/grafana/dashboards
with the new JSON configuration file. -
Restart Grafana Service: To ensure, that all changes take effect, restart the Grafana service. If you’re using Docker Compose, execute the following command:
docker compose restart cvat_grafana
.
For more information, see Grafana Dashboards.
Example of use
This video demonstrates available by default CVAT analytics features.
5 - Mounting cloud storage
AWS S3 bucket as filesystem
Ubuntu 20.04
Mount
-
Install s3fs:
-
Enter your credentials in a file
${HOME}/.passwd-s3fs
and set owner-only permissions: -
Uncomment
user_allow_other
in the/etc/fuse.conf
file:sudo nano /etc/fuse.conf
-
Run s3fs, replace
bucket_name
,mount_point
:
For more details see here.
Automatically mount
Follow the first 3 mounting steps above.
Using fstab
-
Create a bash script named aws_s3_fuse(e.g in /usr/bin, as root) with this content (replace
user_name
on whose behalf the disk will be mounted,backet_name
,mount_point
,/path/to/.passwd-s3fs
): -
Give it the execution permission:
-
Edit
/etc/fstab
adding a line like this, replacemount_point
):
Using systemd
-
Create unit file
sudo nano /etc/systemd/system/s3fs.service
(replaceuser_name
,bucket_name
,mount_point
,/path/to/.passwd-s3fs
): -
Update the system configurations, enable unit autorun when the system boots, mount the bucket:
Check
A file /etc/mtab
contains records of currently mounted filesystems.
Unmount filesystem
If you used systemd to mount a bucket:
Microsoft Azure container as filesystem
Ubuntu 20.04
Mount
-
Set up the Microsoft package repository.(More here)
-
Install
blobfuse
andfuse
:For more details see here
-
Create environments (replace
account_name
,account_key
,mount_point
): -
Create a folder for cache:
-
Make sure the file must be owned by the user who mounts the container:
-
Create the mount point, if it doesn’t exists:
-
Uncomment
user_allow_other
in the/etc/fuse.conf
file:sudo nano /etc/fuse.conf
-
Mount container(replace
your_container
):
Automatically mount
Follow the first 7 mounting steps above.
Using fstab
-
Create configuration file
connection.cfg
with same content, change accountName, select one from accountKey or sasToken and replace with your value: -
Create a bash script named
azure_fuse
(e.g in /usr/bin, as root) with content below (replaceuser_name
on whose behalf the disk will be mounted,mount_point
,/path/to/blobfusetmp
,/path/to/connection.cfg
): -
Give it the execution permission:
-
Edit
/etc/fstab
with the blobfuse script. Add the following line(replace paths):
Using systemd
-
Create unit file
sudo nano /etc/systemd/system/blobfuse.service
. (replaceuser_name
,mount_point
,container_name
,/path/to/connection.cfg
): -
Update the system configurations, enable unit autorun when the system boots, mount the container:
Or for more detail see here
Check
A file /etc/mtab
contains records of currently mounted filesystems.
Unmount filesystem
If you used systemd to mount a container:
If you have any mounting problems, check out the answers to common problems
Google Drive as filesystem
Ubuntu 20.04
Mount
To mount a google drive as a filesystem in user space(FUSE) you can use google-drive-ocamlfuse To do this follow the instructions below:
-
Install google-drive-ocamlfuse:
-
Run
google-drive-ocamlfuse
without parameters:This command will create the default application directory (~/.gdfuse/default), containing the configuration file config (see the wiki page for more details about configuration). And it will start a web browser to obtain authorization to access your Google Drive. This will let you modify default configuration before mounting the filesystem.
Then you can choose a local directory to mount your Google Drive (e.g.: ~/GoogleDrive).
-
Create the mount point, if it doesn’t exist(replace mount_point):
-
Uncomment
user_allow_other
in the/etc/fuse.conf
file:sudo nano /etc/fuse.conf
-
Mount the filesystem:
Automatically mount
Follow the first 4 mounting steps above.
Using fstab
-
Create a bash script named gdfuse(e.g in /usr/bin, as root) with this content (replace
user_name
on whose behalf the disk will be mounted,label
,mount_point
): -
Give it the execution permission:
-
Edit
/etc/fstab
adding a line like this, replacemount_point
):For more details see here
Using systemd
-
Create unit file
sudo nano /etc/systemd/system/google-drive-ocamlfuse.service
. (replaceuser_name
,label
(defaultlabel=default
),mount_point
): -
Update the system configurations, enable unit autorun when the system boots, mount the drive:
For more details see here
Check
A file /etc/mtab
contains records of currently mounted filesystems.
Unmount filesystem
If you used systemd to mount a drive:
6 - LDAP Backed Authentication
The creation of settings.py
When integrating LDAP login, we need to create an overlay to the default CVAT settings located in cvat/settings/production.py. This overlay is where we will configure Django to connect to the LDAP server.
The main issue with using LDAP is that different LDAP implementations have different parameters. So the options used for Active Directory backed authentication will differ if you were to be using FreeIPA.
Update docker-compose.override.yml
In your override config you need to passthrough your settings and tell CVAT to
use them by setting the DJANGO_SETTINGS_MODULE
variable.
Active Directory Example
The following example should allow for users to authenticate themselves against
Active Directory. This example requires a dummy user named cvat_bind
. The
configuration for the bind account does not need any special permissions.
When updating AUTH_LDAP_BIND_DN
, you can write out the account info in two
ways. Both are documented in the config below.
This config is known to work with Windows Server 2022, but should work for older versions and Samba’s implementation of Active Directory.
FreeIPA Example
The following example should allow for users to authenticate themselves against
FreeIPA. This example requires a dummy user named cvat_bind
. The configuration
for the bind account does not need any special permissions.
When updating AUTH_LDAP_BIND_DN
, you can only write the user info in one way,
unlike with Active Directory
This config is known to work with AlmaLinux 8, but may work for other versions and flavors of Enterprise Linux.
Resources
- Microsoft - LDAP Distinguished Names
- Elements that make up a distinguished name. Used with user/group searches.
- Django LDAP Reference Manual
- Other options that can be used for LDAP authentication in Django.
- Django LDAP guide using Active Directory (Unofficial)
- This is not specific to CVAT but can provide insight about firewall rules.
7 - Backup guide
About CVAT data volumes
Docker volumes are used to store all CVAT data:
-
cvat_db
: PostgreSQL database files, used to store information about users, tasks, projects, annotations, etc. Mounted intocvat_db
container by/var/lib/postgresql/data
path. -
cvat_data
: used to store uploaded and prepared media data. Mounted intocvat
container by/home/django/data
path. -
cvat_keys
: used to store the Django secret key. Mounted intocvat
container by/home/django/keys
path. -
cvat_logs
: used to store logs of CVAT backend processes managed by the supervisord service. Mounted intocvat
container by/home/django/logs
path. -
cvat_events_db
: this volume is used to store Clickhouse database files. Mounted intocvat_clickhouse
container by/var/lib/clickhouse
path.
How to backup all CVAT data
All CVAT containers should be stopped before backup:
Please don’t forget to include all the compose config files that were used in the docker compose
command
using the -f
parameter.
Backup data:
Make sure the backup archives have been created, the output of ls backup
command should look like this:
How to restore CVAT from backup
Warning: use exactly the same CVAT version to restore DB. Otherwise it will not work because between CVAT releases the layout of DB can be changed. You always can upgrade CVAT later. It will take care to migrate your data properly internally.
Note: CVAT containers must exist (if no, please follow the installation guide). Stop all CVAT containers:
Restore data:
After that run CVAT as usual:
Additional resources
8 - Upgrade guide
Upgrade guide
Note: updating CVAT from version 2.2.0 to version 2.3.0 requires additional manual actions with database data due to upgrading PostgreSQL base image major version. See details here
To upgrade CVAT, follow these steps:
-
It is highly recommended backup all CVAT data before updating, follow the backup guide and backup all CVAT volumes.
-
Go to the previously cloned CVAT directory and stop all CVAT containers with:
If you have included additional components, include all compose configuration files that are used, e.g.:
-
Update CVAT source code by any preferable way: clone with git or download zip file from GitHub. Note that you need to download the entire source code, not just the Docker Compose configuration file. Check the installation guide for details.
-
Verify settings: The installation process is changed/modified from version to version and you may need to export some environment variables, for example CVAT_HOST.
-
Update local CVAT images. Pull or build new CVAT images, see How to pull/build/update CVAT images section for details.
-
Start CVAT with:
When CVAT starts, it will upgrade its DB in accordance with the latest schema. It can take time especially if you have a lot of data. Please do not terminate the migration and wait till the process is complete. You can monitor the startup process with the following command:
How to upgrade CVAT from v2.2.0 to v2.3.0.
Step by step commands how to upgrade CVAT from v2.2.0 to v2.3.0. Let’s assume that you have CVAT v2.2.0 working.
How to upgrade CVAT from v1.7.0 to v2.2.0.
Step by step commands how to upgrade CVAT from v1.7.0 to v2.2.0. Let’s assume that you have CVAT v1.7.0 working.
How to upgrade PostgreSQL database base image
-
It is highly recommended backup all CVAT data before updating, follow the backup guide and backup CVAT database volume.
-
Run previously used CVAT version as usual
-
Backup current database with
pg_dumpall
tool: -
Stop CVAT:
-
Delete current PostgreSQL’s volume, that’s why it’s important to have a backup:
-
Update CVAT source code by any preferable way: clone with git or download zip file from GitHub. Check the installation guide for details.
-
Start database container only:
-
Import PostgreSQL dump into new DB container:
-
Start CVAT:
9 - Webhooks
Webhooks are user-defined HTTP callbacks that are triggered by specific events. When an event that triggers a webhook occurs, CVAT makes an HTTP request to the URL configured for the webhook. The request will include a payload with information about the event.
CVAT, webhooks can be triggered by a variety of events, such as the creation, deletion, or modification of tasks, jobs, and so on. This makes it easy to set up automated processes that respond to changes made in CVAT.
For example, you can set up webhooks to alert you when a job’s assignee is changed or when a job/task’s status is updated, for instance, when a job is completed and ready for review or has been reviewed. New task creation can also trigger notifications.
These capabilities allow you to keep track of progress and changes in your CVAT workflow instantly.
In CVAT you can create a webhook for a project or organization. You can use CVAT GUI or direct API calls.
See:
- Create Webhook
- Payloads
- Webhook secret
- Ping Webhook
- Webhooks with API calls
- Example of setup and use
Create Webhook
For project
To create a webhook for Project, do the following:
-
Go to the Projects and click on the project’s widget.
-
In the top right corner, click Actions > Setup Webhooks.
-
In the top right corner click +
-
Fill in the Setup webhook form and click Submit.
For organization
To create a webhook for Organization, do the following:
- Create Organization
- Go to the Organization > Settings > Actions > Setup Webhooks.
- In the top right corner click +
- Fill in the Setup webhook form and click Submit.
Webhooks forms
The Setup a webhook forms look like the following.
Forms have the following fields:
Field | Description |
---|---|
Target URL | The URL where the event data will be sent. |
Description | Provides a brief summary of the webhook’s purpose. |
Project | A drop-down list that lets you select from available projects. |
Content type | Defines the data type for the payload in the webhook request via the HTTP Content-Type field. |
Secret | A unique key for verifying the webhook’s origin, ensuring it’s genuinely from CVAT. For more information, see Webhook secret |
Enable SSL | A checkbox for enabling or disabling SSL verification. |
Active | Uncheck this box if you want to stop the delivery of specific webhook payloads. |
Send everything | Check this box to send all event types through the webhook. |
Specify individual events | Choose this option to send only certain event types. Refer to the List of available events for more information on event types. |
List of events
The following events are available for webhook alerts.
Resource | Create | Update | Delete | Description |
---|---|---|---|---|
Organization | ✅ | Alerts for changes made to an Organization. | ||
Membership | ✅ | ✅ | Alerts when a member is added to or removed from an organization. | |
Invitation | ✅ | ✅ | Alerts when an invitation to an Organization is issued or revoked. | |
Project | ✅ | ✅ | ✅ | Alerts for any actions taken within a project. |
Task | ✅ | ✅ | ✅ | Alerts for actions related to a task, such as status changes, assignments, etc. |
Job | ✅ | Alerts for any updates made to a job. | ||
Issue | ✅ | ✅ | ✅ | Alerts for any activities involving issues. |
Comment | ✅ | ✅ | ✅ | Alerts for actions involving comments, such as creation, deletion, or modification. |
Payloads
Create event
Webhook payload object for create:<resource>
events:
Key | Type | Description |
---|---|---|
event |
string |
Identifies the event that triggered the webhook, following the create:<resource> pattern. |
<resource> |
object |
Complete information about the created resource. Refer to the Swagger docs for individual resource details. |
webhook_id |
integer |
The identifier for the webhook that sends the payload. |
sender |
object |
Details about the user that triggered the webhook. |
An example of payload for the create:task
event:
Update event
Webhook payload object for update:<resource>
events:
Key | Type | Description |
---|---|---|
event |
string |
Identifies the event that triggered the webhook, following the update:<resource> pattern. |
<resource> |
object |
Provides complete information about the updated resource. See the Swagger docs for resource details. |
before_update |
object |
Contains keys of <resource> that were updated, along with their old values. |
webhook_id |
integer |
The identifier for the webhook that dispatched the payload. |
sender |
object |
Details about the user that triggered the webhook. |
An example of update:<resource>
event:
Delete event
Webhook payload object for delete:<resource>
events:
Key | Type | Description |
---|---|---|
event |
string |
Identifies the event that triggered the webhook, following the delete:<resource> pattern. |
<resource> |
object |
Provides complete information about the deleted resource. See the Swagger docs for resource details. |
webhook_id |
integer |
The identifier for the webhook that dispatched the payload. |
sender |
object |
Details about the user that triggered the webhook. |
Here is an example of the payload for the delete:task
event:
Webhook secret
To validate that the webhook requests originate from CVAT, include a secret
during the webhook creation process.
When a secret
is provided for the webhook, CVAT includes an X-Signature-256
in the request header of the webhook.
CVAT uses the SHA256 hash function to encode the request body for the webhook and places the resulting hash into the header.
The webhook recipient can verify the source of the request
by comparing the received X-Signature-256
value with the expected value.
Here’s an example of a header value for a request with an empty body and secret = mykey
:
X-Signature-256: e1b24265bf2e0b20c81837993b4f1415f7b68c503114d100a40601eca6a2745f
Here is an example of how you can verify a webhook signature in your webhook receiver service:
Ping Webhook
To confirm the proper configuration of your webhook and ensure that CVAT can establish a connection with the target URL, use the Ping webhook feature.
- Click the Ping button in the user interface (or send a
POST /webhooks/{id}/ping
request through API). - CVAT will send a webhook alert to the specified target URL with basic information about the webhook.
Ping webhook payload:
Key | Type | Description |
---|---|---|
event |
string |
The value is always ping . |
webhook |
object |
Complete information about the webhook. See the Swagger docs for a detailed description of fields. |
sender |
object |
Information about the user who initiated the ping on the webhook. |
Here is an example of a payload for the ping
event:
Webhooks with API calls
To create webhook via an API call, see Swagger documentation.
For examples, see REST API tests.
Example of setup and use
This video demonstrates setting up email alerts for a project using Zapier and Gmail.
10 - Custom Certificates
CVAT use traefik as a reverse proxy to manage SSL certificates. By default, traefik uses Let’s Encrypt to generate SSL certificates. However, you can use your own certificates instead of Let’s Encrypt.
See:
Setup Custom Certificates
Create Certificates Directory
Create a certs
directory in the root of the project:
Move your certificates to the ./certs
directory:
Change Traefik Configuration
Create tls.yml
in the root of the project directory with the following content:
Edit the docker-compose.https.yml
file and change the traefik service configuration as follows:
Start CVAT
Start CVAT with the following command: