CVAT Analytics and monitoring

CVAT Analytics suite of tools is designed to track and understand users’ behavior, system performance, and for identifying potential issues in your application.

You can also visualize user activity through Grafana, and aggregate user working time by the jobs.

Gathered logs can be additionally filtered for efficient debugging.

By using analytics, you’ll gain valuable insights to optimize your system and enhance user satisfaction.

CVAT analytics are available from the top menu.

Superusers and users with administrator role have access to analytics. Permission to access analytics can also be granted when editing a user on admin page by Has access to analytics checkbox.

Note: CVAT analytics and monitoring are available only for on-prem solution.

See:

• High-level architecture
• CVAT Analytics
  • Ports settings
  • Events log structure
  • Types of supported events
  • Working time calculation
  • Request id for tracking
  • Fetching event data as CSV from the /api/events endpoint
• Dashboards
  • Dashboard: All Events
  • Dashboard: Management
  • Dashboard: Monitoring
  • Dashboards setup
• Example of use

High-level architecture

The CVAT analytics is based on Vector, ClickHouse, and Grafana.

CVAT Analytics

CVAT and its analytics module can be set up locally, for self-hosted solution analytics are enabled by default.

For detailed CVAT installation instructions, see Installation Guide or refer to the CVAT Course for installation videos.

All analytics-related features will be launched when you start CVAT containers with the following command:

docker compose up -d

Ports settings

If you cannot access analytics on development environnement, see Analytics Ports

Events log structure

Relational database schema with the following fields:

Types of supported events

Supported events change the scope of information displayed in Grafana.

Server events:

• create:project, update:project, delete:project

• create:task, update:task, delete:task

• create:job, update:job, delete:job

• create:organization, update:organization, delete:organization

• create:user, update:user, delete:user

• create:cloudstorage, update:cloudstorage, delete:cloudstorage

• create:issue, update:issue, delete:issue

• create:comment, update:comment, delete:comment

• create:annotations, update:annotations, delete:annotations

• create:label, update:label, delete:label

• export:dataset, import:dataset

• call:function

Client events:

• load:cvat

• load:job, save:job

• send:exception

• draw:object, paste:object, copy:object, propagate:object, drag:object, resize:object, delete:object, merge:objects, split:objects, group:objects, slice:object, join:objects

• change:frame

• zoom:image, fit:image, rotate:image

• action:undo, action:redo

• run:annotations_action

• click:element

• debug:info

Working time calculation

Here is a short overview of how CVAT deals with the user’s working time:

• The user interface collects events when a user interacts with the interface (resizing canvas, drawing objects, clicking buttons, etc) The structure of one single event is described here.

• The user interface sends these events in bulk to the server. Currently, it uses the following triggers to send events:

  • Periodical timer (~90 seconds)
  • A user clicks the “Save” button on the annotation view
  • A user opens the annotation view
  • A user closes the annotation view (but not the tab/browser)
  • A user clicks Logout button

• When events reach the server, it calculates working time based on timestamps of the events.

• The working time for an event is computed as the sum of the following:

  • The difference between the start time of the event and the end time of the previous event, if it is not more than 100 seconds.
  • The duration of the event, for events of type change:frame.

• After calculation, the server generates send:working_time events with time value in payload. These events may or may not be bound to a certain job/task/project, depending on the client-side events that were used to generate them.

• CVAT saves the event in the database and later these events are used to compute metrics for analytics.

Request id for tracking

Note, that every response to an API request made to the the server includes a header named X-Request-Id, for example: X-Request-Id: 6a2b7102-c4b9-4d57-8754-5658132ba37d.

This identifier is also recorded in all server events that occur as a result of the respective request.

For example, when an operation to create a task is performed, other related entities such as labels and attributes are generated on the server in addition to the Task object.

All events associated with this operation will have the same request_id in the payload field.

Fetching event data as CSV from the /api/events endpoint

The /api/events endpoint allows the fetching of event data with filtering parameters such as org_id, project_id, task_id, job_id, and user_id.

For more details, see Swagger API Documentation.

For example, to fetch all events associated with a specific job, the following curl command can be used:

curl --user 'user:pass' https://app.cvat.ai/api/events?job_id=123

In the response, you will receive a query ID:

{ "query_id": "150cac1f-09f1-4d73-b6a5-5f47aa5d0031" }

As this process may take some time to complete, the status of the request can be checked by adding the query parameter query_id to the request:

curl -I --user 'user:pass' https://app.cvat.ai/api/events?job_id=123&query_id=150cac1f-09f1-4d73-b6a5-5f47aa5d0031

Upon successful creation, the server will return a 201 Created status:

HTTP/2 201
allow: GET, POST, HEAD, OPTIONS
date: Tue, 16 May 2023 13:38:42 GMT
referrer-policy: same-origin
server: Apache
vary: Accept,Origin,Cookie
x-content-type-options: nosniff
x-frame-options: DENY
x-request-id: 4631f5fa-a4f0-42a8-b77b-7426fc298a85

The CSV file can be downloaded by adding the action=download query parameter to the request:

curl --user 'user:pass' https://app.cvat.ai/api/events?job_id=123&query_id=150cac1f-09f1-4d73-b6a5-5f47aa5d0031&action=download > /tmp/events.csv

This will download and save the file to /tmp/events.csv on your local machine.

Dashboards

By default, three dashboards are available in CVAT.

To access them, click General, you will be forwarded to the page with available dashboards.

Dashboard: All Events

The dashboard shows all events, their timestamps, and their source.

There are two fields with statistics at the bottom of the dashboard, about browser and OS users use.

Click on the column name to enable a filter.

If you want to inspect the value, hover over it and click on the eye icon.

Dashboard: Management

The dashboard shows user activity.

Click on the column name to enable a filter.

If you want to inspect the value, hover over it and click on the eye icon.

Dashboard: Monitoring

The dashboard shows server logs, helps handle errors, and shows user activity.

Click on the column name to enable a filter.

If you want to inspect the value, hover over it and click on the eye icon.

Dashboards setup

You can adjust the dashboards. To do this, click on the graph or table name and from the drop-down menu select Edit.

Adjust the query in the editor.

Example of query:

SELECT
    time,
    uniqExact(user_id) Users
FROM
(
    SELECT
      user_id,
      toStartOfInterval(timestamp, INTERVAL 15 minute) as time
    FROM cvat.events
    WHERE
      user_id IS NOT NULL
    GROUP BY
      user_id,
      time
    ORDER BY time ASC WITH FILL STEP toIntervalMinute(15)
)
GROUP BY time
ORDER BY time

Note, that by default the updated configuration will not be saved and will be reset to the default parameters after you restart the container.

To save the updated configuration, do the following:

1. Update Configuration: Start by making your desired changes in the query.

2. Apply Changes: Once you’ve made your changes, click the Apply button to ensure the changes are implemented.

   

3. Save Configuration: To save your applied changes, on the top of the dashboard, click the Save button.

   

4. Replace Configuration File: After saving, replace the existing Grafana dashboard configuration file is located at components/analytics/grafana/dashboards with the new JSON configuration file.

   

5. Restart Grafana Service: To ensure, that all changes take effect, restart the Grafana service. If you’re using Docker Compose, execute the following command: docker compose restart cvat_grafana.

For more information, see Grafana Dashboards.

Example of use

This video demonstrates available by default CVAT analytics features.