Upgrade guide

Relevant for:CVAT Community

Upgrade guide

Note: updating CVAT from version 2.2.0 to version 2.3.0 requires additional manual actions with database data due to upgrading PostgreSQL base image major version. See details here

To upgrade CVAT, follow these steps:

  • It is highly recommended backup all CVAT data before updating, follow the backup guide and backup all CVAT volumes.

  • Go to the previously cloned CVAT directory and stop all CVAT containers with:

    docker compose down

    If you have included additional components, include all compose configuration files that are used, e.g.:

    docker compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml down

  • Update CVAT source code by any preferable way: clone with git or download zip file from GitHub. Note that you need to download the entire source code, not just the Docker Compose configuration file. Check the installation guide for details.

  • Verify settings: The installation process is changed/modified from version to version and you may need to export some environment variables, for example CVAT_HOST.

  • Update local CVAT images. Pull or build new CVAT images, see How to pull/build/update CVAT images section for details.

  • Start CVAT with:

    docker compose up -d

    When CVAT starts, it will upgrade its DB in accordance with the latest schema. It can take time especially if you have a lot of data. Please do not terminate the migration and wait till the process is complete. You can monitor the startup process with the following command:

    docker logs cvat_server -f

Upgrade to v2.72.0 or later

Version 2.72.0 moves CVAT application source files from /home/django to /opt/cvat. If your deployment bind-mounts files into the application source tree, update their container paths before starting the new version. For example, mount a custom settings module at /opt/cvat/cvat_enterprise/settings/custom_settings.py and a replacement logo at /opt/cvat/cvat/apps/engine/static/logo.svg.

The CVAT data directories and auth_config.yml are not application source files: they remain under /home/django (for example, /home/django/auth_config.yml).

How to upgrade CVAT from v2.46.0 to v2.47.0.

In version 2.47.0, CVAT upgraded the FFmpeg library it uses to split videos into frames from 4.3.1 to 8.0. There is a small chance that some video files may not be processed differently by the new FFmpeg version.

If one of your tasks is affected, follow the guide in ./utils/ffmpeg_compatibility/README.md

Upgrade CVAT after v2.26.0

In version 2.26.0, CVAT changed the location where the export cache is stored. To clean up the outdated cache, run the command depending on how CVAT is deployed:

  docker exec -it cvat_server python manage.py cleanuplegacyexportcache
  
  cvat_backend_pod=$(kubectl get pods -l component=server -o 'jsonpath={.items[0].metadata.name}')
  kubectl exec -it ${cvat_backend_pod} -- python manage.py cleanuplegacyexportcache
  
  python manage.py cleanuplegacyexportcache

How to upgrade CVAT from v2.2.0 to v2.3.0.

Step by step commands how to upgrade CVAT from v2.2.0 to v2.3.0. Let’s assume that you have CVAT v2.2.0 working.

docker exec -it cvat_db pg_dumpall > cvat.db.dump
cd cvat
docker compose down
docker volume rm cvat_cvat_db
export CVAT_VERSION="v2.3.0"
cd ..
mv cvat cvat_220
wget https://github.com/cvat-ai/cvat/archive/refs/tags/${CVAT_VERSION}.zip
unzip ${CVAT_VERSION}.zip && mv cvat-${CVAT_VERSION:1} cvat
unset CVAT_VERSION
cd cvat
export CVAT_HOST=cvat.example.com
export ACME_EMAIL=example@example.com
docker compose pull
docker compose up -d cvat_db
docker exec -i cvat_db psql -q -d postgres < ../cvat.db.dump
docker compose -f docker-compose.yml -f docker-compose.dev.yml -f docker-compose.https.yml up -d

How to upgrade CVAT from v1.7.0 to v2.2.0.

Step by step commands how to upgrade CVAT from v1.7.0 to v2.2.0. Let’s assume that you have CVAT v1.7.0 working.

export CVAT_VERSION="v2.2.0"
cd cvat
docker compose down
cd ..
mv cvat cvat_170
wget https://github.com/cvat-ai/cvat/archive/refs/tags/${CVAT_VERSION}.zip
unzip ${CVAT_VERSION}.zip && mv cvat-${CVAT_VERSION:1} cvat
cd cvat
docker pull cvat/server:${CVAT_VERSION}
docker tag cvat/server:${CVAT_VERSION} openvino/cvat_server:latest
docker pull cvat/ui:${CVAT_VERSION}
docker tag cvat/ui:${CVAT_VERSION} openvino/cvat_ui:latest
docker compose up -d

How to upgrade PostgreSQL database base image

  1. It is highly recommended backup all CVAT data before updating, follow the backup guide and backup CVAT database volume.

  2. Run previously used CVAT version as usual

  3. Backup current database with pg_dumpall tool:

    docker exec -it cvat_db pg_dumpall > cvat.db.dump

  4. Stop CVAT:

    docker compose down

  5. Delete current PostgreSQL’s volume, that’s why it’s important to have a backup:

    docker volume rm cvat_cvat_db

  6. Update CVAT source code by any preferable way: clone with git or download zip file from GitHub. Check the installation guide for details.

  7. Start database container only:

    docker compose up -d cvat_db

  8. Import PostgreSQL dump into new DB container:

    docker exec -i cvat_db psql -q -d postgres < cvat.db.dump

  9. Start CVAT:

    docker compose up -d

Migrate Redis from Bitnami to CloudPirates

This procedure applies to CVAT installations deployed with the Helm chart. It migrates a standalone Redis instance from the Bitnami chart to the CloudPirates chart.

The migration requires downtime. Do not start it while imports, exports, backups, or other background jobs are running.

There are two ways to transfer the Redis data:

  • Reuse the existing persistent volume claim (PVC). Use this method when the old and new Redis instances run in the same Kubernetes cluster and the existing volume can be attached to the new pod.
  • Export and restore an RDB file. Use this method when a new PVC is required or when the storage cannot be reused directly.

Before you begin

  1. Confirm that the current Redis deployment uses the standalone architecture. This procedure does not cover Redis replication, Sentinel, or Redis Cluster.
  2. Keep the Redis server version unchanged during the chart migration. Upgrade Redis itself in a separate maintenance operation after the chart migration has been completed and data migration is verified.
  3. Check that the node or volume has enough free space for an RDB snapshot. (If you have chosen RDB dump method)

This guide uses the following environment variables:

export CVAT_NAMESPACE="cvat"
export CVAT_RELEASE="cvat"
export REDIS_POD_NAME="cvat-redis-master-0"

Put CVAT into maintenance mode

Scale the CVAT backend and frontend deployments to zero before upgrading the Helm release. At this point the release must still use Bitnami Redis.

So we have added values-maintenance.yaml file to the chart that will scale down CVAT writers. Please be sure that it is added last in a chain values files.

helm -n $CVAT_NAMESPACE upgrade $CVAT_RELEASE path_to_chart -f your_values_file -f values-maintenance.yaml

ex: helm -n cvat upgrade cvat . -f myvalues.yaml -f values-maintenance.yaml

The Bitnami Redis pod must remain running.

kubectl get pods --namespace "$CVAT_NAMESPACE"

kubectl exec --namespace "$CVAT_NAMESPACE" -it $REDIS_POD_NAME -- /bin/bash

ex. kubectl -n cvat exec -it cvat-redis-master-0 – /bin/bash

Login to the Redis cli with your Redis password (you can find Redis password in your Redis secret)

redis-cli -a your_password_for_redis

Record the database size and keyspace information. DBSIZE reports the number of keys in the currently selected database; INFO keyspace reports all non-empty databases.

DBSIZE
INFO keyspace

Create the final RDB snapshot, just run this command in the Redis cli:

SAVE
exit
exit

The command must return OK. Do not continue if it fails.

Copy dump locally, even if you will be using PVC method, just in case.

kubectl cp \
  "$CVAT_NAMESPACE/$REDIS_POD_NAME:/data/dump.rdb" \
  ./cvat-redis-dump.rdb

Method 1: Reuse the existing PVC

Find the PVC mounted as the Bitnami Redis data volume:

OLD_REDIS_PVC=$(kubectl get pod \
  --namespace "$CVAT_NAMESPACE" \
  $REDIS_POD_NAME \
  -o jsonpath='{.spec.volumes[?(@.name=="redis-data")].persistentVolumeClaim.claimName}')

echo "$OLD_REDIS_PVC"

Do not continue if the command returns an empty value.

Find the persistent volume and check its reclaim policy:

REDIS_PV=$(kubectl get pvc \
  --namespace "$CVAT_NAMESPACE" \
  "$OLD_REDIS_PVC" \
  -o jsonpath='{.spec.volumeName}')

kubectl get pv "$REDIS_PV" \
  -o jsonpath='{.spec.persistentVolumeReclaimPolicy}{"\n"}'

If the policy is Delete, change it to Retain before replacing the StatefulSet:

kubectl patch pv "$REDIS_PV" \
  --type merge \
  -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Add the existing claim to your Helm chart values file.

redis:
  persistence:
    existingClaim: "<existing-redis-pvc>"

The existingClaim setting must remain in the values file after the migration. Removing it causes the CloudPirates StatefulSet to use a different PVC.

If you want to update file permissions on the PVC, because Bitnami and CloudPirates use different user IDs, enable the CloudPirates volume-permissions init container, provided that the cluster security policy permits it:

redis:
  volumePermissions:
    enabled: true

Continue with Prepare and install the new chart.

Method 2: Transfer an RDB dump

Connect to the Redis pod:

kubectl exec --namespace "$CVAT_NAMESPACE" -it $REDIS_POD_NAME -- /bin/bash

Login to the Redis cli with your Redis password (you can find Redis password in your Redis secret)

redis-cli -a your_password_for_redis

Confirm the source data directory:

CONFIG GET dir
exit
exit

The standard Bitnami configuration returns /data.

Copy the snapshot to a machine with enough free disk space:

kubectl cp \
  "$CVAT_NAMESPACE/$REDIS_POD_NAME:/data/dump.rdb" \
  ./cvat-redis-dump.rdb

Keep this file until the migration and the CVAT upgrade have been verified, then continue with the next section.

Prepare and install the new chart

Ensure that redis service name override is empty. This setting was added to ensure Bitnami Redis compatibility, however for CloudPirates Redis the name of the service is generated by Helm helper function, so no override is needed.

cvat:
  backend:
    redisInmemHostOverride: ""

In the target CVAT version, Chart.yaml must refer to the CloudPirates chart:

- name: redis
  version: "0.32.4"
  repository: https://cloudpirates-io.github.io/helm-charts
  condition: redis.enabled

Update the chart dependencies and check the result:

cd your_helm_chart_directory
helm dependency update
helm dependency list

The dependency list must contain one redis entry with the CloudPirates repository and an ok status. An incorrect version status means that Chart.lock or the Redis archive in the charts directory does not match Chart.yaml.

Use the same Redis password secret and secret key that were used by the Bitnami deployment. For the standard CVAT chart, the relevant settings are:

redis:
  auth:
    existingSecret: "{{ .Release.Name }}-redis-secret"
    existingSecretPasswordKey: password

Install the new chart while keeping CVAT in maintenance mode. Include every values file used by the existing release, in the same order, followed by the values-maintenance.yaml:

cd your_cvat_helm_chart_directory

helm upgrade \
  --namespace "$CVAT_NAMESPACE" \
  "$CVAT_RELEASE" \
   . \
  -f "your-cvat-values.yaml" \
  -f "values-maintenance.yaml"

Wait for the CloudPirates Redis pod to become ready.

For the PVC method, continue with Verify the migrated data.

For the RDB method, restore the dump before verification.

Restore the RDB dump

This section applies only to the RDB method.

Connect to the new Redis pod and check its persistence settings:

NEW_REDIS_POD="$CVAT_RELEASE-redis-0"

kubectl exec --namespace "$CVAT_NAMESPACE" "$NEW_REDIS_POD" -it -- /bin/bash

redis-cli -a your_password_for_redis

CONFIG GET dir

CONFIG GET appendonly
exit
exit

Continue only if the data directory is /data and appendonly is no. When AOF is enabled, Redis does not restore its state from the RDB file in this procedure. (By default CloudPirates Redis does not use AOF)

Copy the saved snapshot into the new pod:

kubectl cp \
  ./cvat-redis-dump.rdb \
  "$CVAT_NAMESPACE/$NEW_REDIS_POD:/data/dump.rdb"

Restart Redis without saving the empty database that is currently in memory:

kubectl exec --namespace "$CVAT_NAMESPACE" "$NEW_REDIS_POD" -- \
  env REDISCLI_AUTH="your_redis_password" redis-cli SHUTDOWN NOSAVE

The connection closes when Redis stops. Kubernetes restarts the container, and Redis loads /data/dump.rdb during startup.

Redis might overwrite your backup when it receives SIGTERM signal from Kubernetes, so we need to force it not to save its state during shutdown. This is done by using SHUTDOWN NOSAVE command.

Wait for the pod to become ready again.

Verify the migrated data

Check the database size and keyspace information in the new Redis instance:

NEW_REDIS_POD="$CVAT_RELEASE-redis-0"

kubectl exec --namespace "$CVAT_NAMESPACE" "$NEW_REDIS_POD" -it -- /bin/bash

redis-cli -a your_password_for_redis

DBSIZE

INFO keyspace

Compare the output with the values recorded before the migration. Expiring keys can make the counts slightly lower. Investigate large differences before bringing CVAT back online.

Use SCAN for an optional key check:

SCAN 0 MATCH "rq:*" COUNT 100

Bring CVAT back online

Change redisInmemHostOverride to the empty value in the values.yaml file. Otherwise, the backend will not be able to connect to the new Redis instance. Please remember that last values file in the chain has the highest priority.

cvat:
  backend:
    redisInmemHostOverride: ""

Run the upgrade again without values-maintenance.yaml. Keep the migration values file in the command:

cd your_cvat_helm_chart_directory

helm upgrade \
  --namespace "$CVAT_NAMESPACE" \
  "$CVAT_RELEASE" \
   . \
  -f your_cvat_helm_chart_values.yaml

Wait for the backend and frontend deployments to become ready. Check the backend logs for Redis connection or authentication errors, then verify login and run a small background operation such as an export.

Do not delete the local RDB dump until these checks have passed and the normal backup policy has produced a new verified backup. Old PVC is now used by the new Redis pod if you have chosen the PVC method. If not, you can delete it after data was validated.

Things to notice

  1. Bitnami Redis uses a different user ID than CloudPirates Redis. It might cause issues and can be fixed with volumePermissions.enabled: true.
  2. Bitnami Redis uses AOF, while CloudPirates Redis does not. AOF can be enabled using Helm values after the migration is complete.
  3. Do not delete backups immediately after migration.

Migrate PostgreSQL from Bitnami to CloudPirates

This procedure applies to CVAT installations deployed with the Helm chart. It migrates the CVAT database from the Bitnami PostgreSQL chart to the CloudPirates PostgreSQL chart by creating a logical dump and restoring it into a new persistent volume claim (PVC).

The migration requires downtime. Do not start it while imports, exports, backups, or other background jobs are running.

Before you begin

  1. Back up all CVAT data as described in the backup guide.
  2. Keep the PostgreSQL major version unchanged during the chart migration. The example configuration migrates PostgreSQL 15 to PostgreSQL 15. Upgrade the database major version in a separate maintenance operation.
  3. Check the source database size and make sure that both the migration PVC and the new PostgreSQL PVC have enough free space. The supplied migration manifest requests 20 GiB; increase it before applying the manifest when necessary.
  4. Record every values file used by the current Helm release and its order. Use the same files in every helm upgrade command below, with values-maintenance.yaml last while maintenance mode is required.
  5. The supplied migration manifest assumes that the Helm release is named cvat. If another release name is used, update PGHOST and the referenced secret name in postgres-migration/pg-dump-pod.yaml before applying it.

This guide uses the following environment variables:

export CVAT_NAMESPACE="cvat"
export CVAT_RELEASE="cvat"
export POSTGRES_STS="${CVAT_RELEASE}-postgresql"
export POSTGRES_POD="${POSTGRES_STS}-0"
export OLD_POSTGRES_PVC="data-${POSTGRES_STS}-0"
export MIGRATION_POD="postgres-migration-dump"

The StatefulSet, pod, and PVC names above are the defaults for the CVAT chart. Confirm them before continuing:

kubectl get statefulset,pod,pvc --namespace "$CVAT_NAMESPACE"

Put CVAT into maintenance mode

Before changing Chart.yaml or the PostgreSQL values, upgrade the existing release with the Bitnami PostgreSQL chart still configured:

cd your_cvat_helm_chart_directory

helm upgrade \
  --namespace "$CVAT_NAMESPACE" \
  "$CVAT_RELEASE" \
  . \
  -f your-cvat-values.yaml \
  -f values-maintenance.yaml

Wait until the CVAT frontend, backend server, initializer, and worker pods have terminated. The Bitnami PostgreSQL pod must remain running and ready. Do not create the final dump until all database writers have stopped.

Create and verify the database dump

Apply the supplied migration manifest:

kubectl apply \
  --namespace "$CVAT_NAMESPACE" \
  -f postgres-migration/pg-dump-pod.yaml

kubectl wait \
  --namespace "$CVAT_NAMESPACE" \
  --for=condition=Ready \
  "pod/$MIGRATION_POD" \
  --timeout=5m

Follow the migration pod logs:

kubectl logs \
  --namespace "$CVAT_NAMESPACE" \
  --follow "$MIGRATION_POD"

Wait for the following message, then stop following the logs with Ctrl+C:

Dump saved to /backup/cvat.dump

Do not continue if pg_dump reports an error. Verify that the custom-format dump is readable and record the source roles, database owner, and tables:

kubectl exec --namespace "$CVAT_NAMESPACE" "$MIGRATION_POD" -- \
  pg_restore --list /backup/cvat.dump > /dev/null

kubectl exec --namespace "$CVAT_NAMESPACE" "$MIGRATION_POD" -- \
  psql -d postgres -c 'SHOW server_version;'

kubectl exec --namespace "$CVAT_NAMESPACE" "$MIGRATION_POD" -- \
  psql -d postgres -c '\du'

kubectl exec --namespace "$CVAT_NAMESPACE" "$MIGRATION_POD" -- \
  psql -d postgres -c '\l'

kubectl exec --namespace "$CVAT_NAMESPACE" "$MIGRATION_POD" -- \
  psql -d cvat -c '\dt'

kubectl exec --namespace "$CVAT_NAMESPACE" "$MIGRATION_POD" -- \
  psql -d cvat -c \
    'SELECT pg_size_pretty(pg_database_size(current_database())) AS database_size;'

The source installation must contain both the postgres and cvat roles, and the cvat role must own the cvat database. Investigate any different setup before continuing because the target chart creates this standard role and ownership layout.

Copy the dump to a machine with enough free disk space. Keep this independent copy even though the migration pod retains another copy on its PVC:

kubectl cp \
  --namespace "$CVAT_NAMESPACE" \
  "$MIGRATION_POD:/backup/cvat.dump" \
  ./cvat.dump

Protect the old PostgreSQL volume

Check the StatefulSet PVC retention policy. An empty result means that the Kubernetes default, Retain, applies:

kubectl get statefulset \
  --namespace "$CVAT_NAMESPACE" \
  "$POSTGRES_STS" \
  -o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy}{"\n"}'

If either whenDeleted or whenScaled is Delete, change both values to Retain before deleting the StatefulSet:

kubectl patch statefulset \
  --namespace "$CVAT_NAMESPACE" \
  "$POSTGRES_STS" \
  --type merge \
  -p '{"spec":{"persistentVolumeClaimRetentionPolicy":{"whenDeleted":"Retain","whenScaled":"Retain"}}}'

Find the persistent volume (PV) bound to the old PVC and check its reclaim policy:

OLD_POSTGRES_PV=$(kubectl get pvc \
  --namespace "$CVAT_NAMESPACE" \
  "$OLD_POSTGRES_PVC" \
  -o jsonpath='{.spec.volumeName}')

kubectl get pv "$OLD_POSTGRES_PV" \
  -o jsonpath='{.spec.persistentVolumeReclaimPolicy}{"\n"}'

If the reclaim policy is Delete, change it to Retain:

kubectl patch pv "$OLD_POSTGRES_PV" \
  --type merge \
  -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

The StatefulSet retention policy protects the PVC when the StatefulSet is deleted. The PV reclaim policy protects the underlying storage if the PVC is deleted later. These are separate safeguards, and both must be checked.

Prepare the CloudPirates chart

In the target CVAT version, Chart.yaml must refer to the CloudPirates chart using the postgresql alias:

- name: postgres
  version: "0.20.5"
  repository: https://cloudpirates-io.github.io/helm-charts
  condition: postgresql.enabled
  alias: postgresql

Update the chart dependencies and check the result:

helm dependency update
helm dependency list

The dependency list must contain a postgres entry for the CloudPirates repository with an ok status.

Replace the Bitnami-specific postgresql values with the CloudPirates values. The important settings for the standard CVAT chart are:

postgresql:
  enabled: true

  image:
    registry: docker.io
    repository: postgres
    tag: "15.19@sha256:9b1d34adbce1dd07ee6e94b4a2cf698884b89bd44a6c9c12f5da8f3acbfe4957"
    imagePullPolicy: IfNotPresent

  auth:
    username: postgres
    database: postgres
    existingSecret: "{{ .Release.Name }}-postgres-secret"
    secretKeys:
      adminPasswordKey: postgres-password

  customUser:
    name: cvat
    database: cvat
    existingSecret: "{{ .Release.Name }}-postgres-secret"
    secretKeys:
      name: username
      database: database
      password: password

  service:
    port: 5432
    targetPort: 5432

  persistence:
    enabled: true
    volumeName: data-cloudpirates
    size: 8Gi

  secret:
    create: true
    name: "{{ .Release.Name }}-postgres-secret"
    password: cvat_postgresql
    postgres_password: cvat_postgresql_postgres
    replication_password: cvat_postgresql_replica

Set persistence.size and, when required, persistence.storageClass for the target cluster. volumeName: data-cloudpirates is intentional: it makes the new StatefulSet create a new PVC instead of mounting the Bitnami data volume. Do not reuse the old PVC because the charts use different data directory layouts and container user IDs.

Keep the existing database passwords or existing secret during this migration. Do not combine the chart migration with credential rotation or a PostgreSQL major-version upgrade. Also translate any custom resources, scheduling, security context, PostgreSQL configuration, and storage settings from the old chart to their CloudPirates equivalents.

Install CloudPirates PostgreSQL

Delete the old StatefulSet, then immediately confirm that its PVC still exists:

kubectl delete statefulset \
  --namespace "$CVAT_NAMESPACE" \
  "$POSTGRES_STS"

kubectl wait \
  --namespace "$CVAT_NAMESPACE" \
  --for=delete \
  "pod/$POSTGRES_POD" \
  --timeout=5m

kubectl get pvc \
  --namespace "$CVAT_NAMESPACE" \
  "$OLD_POSTGRES_PVC"

Do not continue if the old PVC is missing. Install the new chart while keeping CVAT in maintenance mode:

helm upgrade \
  --namespace "$CVAT_NAMESPACE" \
  "$CVAT_RELEASE" \
  . \
  -f your-cvat-values.yaml \
  -f values-maintenance.yaml

Wait for the new PostgreSQL StatefulSet to become ready:

kubectl rollout status \
  --namespace "$CVAT_NAMESPACE" \
  "statefulset/$POSTGRES_STS" \
  --timeout=10m

Confirm that the new PostgreSQL pod uses a new PVC and that the old PVC is still present. With the values above, the new PVC is named data-cloudpirates-<release>-postgresql-0:

kubectl get pod,pvc --namespace "$CVAT_NAMESPACE"

Restore and verify the database

The migration pod now connects to the new PostgreSQL service. Confirm that the target database contains the expected roles and database ownership:

kubectl exec --namespace "$CVAT_NAMESPACE" "$MIGRATION_POD" -- \
  psql -d postgres -c 'SHOW server_version;'

kubectl exec --namespace "$CVAT_NAMESPACE" "$MIGRATION_POD" -- \
  psql -d postgres -c '\du'

kubectl exec --namespace "$CVAT_NAMESPACE" "$MIGRATION_POD" -- \
  psql -d postgres -c '\l'

kubectl exec --namespace "$CVAT_NAMESPACE" "$MIGRATION_POD" -- \
  psql -d cvat -c '\dt'

Both the postgres and cvat roles must exist, the cvat role must own the cvat database, and the new cvat database must not contain application tables before the restore.

Restore the dump as one transaction and stop on the first error:

kubectl exec --namespace "$CVAT_NAMESPACE" "$MIGRATION_POD" -- \
  pg_restore \
    --verbose \
    --exit-on-error \
    --single-transaction \
    --dbname=cvat \
    /backup/cvat.dump

Do not bring CVAT back online if the restore reports an error. After a successful restore, list the restored tables and compare them with the source table list recorded earlier:

kubectl exec --namespace "$CVAT_NAMESPACE" "$MIGRATION_POD" -- \
  psql -d cvat -c '\dt'

Bring CVAT back online

Run the upgrade again without values-maintenance.yaml. Keep every other values file and its order unchanged:

helm upgrade \
  --namespace "$CVAT_NAMESPACE" \
  "$CVAT_RELEASE" \
  . \
  -f your-cvat-values.yaml

Wait for all CVAT workloads to become ready. Check the backend and worker logs for database connection or migration errors, then verify login and perform a small read/write operation such as creating a task and an export.

Keep the local dump, the migration PVC, and the old Bitnami PostgreSQL PVC and PV until all checks have passed and the normal backup policy has produced a new verified backup. The retained old volume is also the rollback path: if the restore or verification fails, keep CVAT in maintenance mode, restore the Bitnami dependency and values, delete the new PostgreSQL StatefulSet, and run the Helm upgrade again so that Bitnami reattaches its original PVC.

After the migration has been fully validated, the migration pod and PVC can be removed:

kubectl delete \
  --namespace "$CVAT_NAMESPACE" \
  -f postgres-migration/pg-dump-pod.yaml

Deleting the old Bitnami PVC after its PV policy was changed to Retain leaves the PV and underlying storage for manual cleanup; remove them only when the old database and rollback path are no longer needed.