This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Advanced

This section contains basic documents for system administrators

1: Analytics for Computer Vision Annotation Tool (CVAT)
2: Semi-automatic and Automatic Annotation
3: Mounting cloud storage
4: Backup guide

1 - Analytics for Computer Vision Annotation Tool (CVAT)

This section on GitHub

It is possible to proxy annotation logs from client to ELK. To do that run the following command below:

Build docker image

# From project root directory
docker-compose -f docker-compose.yml -f components/analytics/docker-compose.analytics.yml build

Run docker container

# From project root directory
docker-compose -f docker-compose.yml -f components/analytics/docker-compose.analytics.yml up -d

At the moment it is not possible to save advanced settings. Below values should be specified manually.

Time picker default

{ “from”: “now/d”, “to”: “now/d”, “display”: “Today”, “section”: 0 }

Time picker quick ranges

[
  {
    "from": "now/d",
    "to": "now/d",
    "display": "Today",
    "section": 0
  },
  {
    "from": "now/w",
    "to": "now/w",
    "display": "This week",
    "section": 0
  },
  {
    "from": "now/M",
    "to": "now/M",
    "display": "This month",
    "section": 0
  },
  {
    "from": "now/y",
    "to": "now/y",
    "display": "This year",
    "section": 0
  },
  {
    "from": "now/d",
    "to": "now",
    "display": "Today so far",
    "section": 2
  },
  {
    "from": "now/w",
    "to": "now",
    "display": "Week to date",
    "section": 2
  },
  {
    "from": "now/M",
    "to": "now",
    "display": "Month to date",
    "section": 2
  },
  {
    "from": "now/y",
    "to": "now",
    "display": "Year to date",
    "section": 2
  },
  {
    "from": "now-1d/d",
    "to": "now-1d/d",
    "display": "Yesterday",
    "section": 1
  },
  {
    "from": "now-1w/w",
    "to": "now-1w/w",
    "display": "Previous week",
    "section": 1
  },
  {
    "from": "now-1m/m",
    "to": "now-1m/m",
    "display": "Previous month",
    "section": 1
  },
  {
    "from": "now-1y/y",
    "to": "now-1y/y",
    "display": "Previous year",
    "section": 1
  }
]

2 - Semi-automatic and Automatic Annotation

This page provides information about the installation of components needed for semi-automatic and automatic annotation

⚠ WARNING: Do not use docker-compose up If you did, make sure all containers are stopped by docker-compose down.

To bring up cvat with auto annotation tool, from cvat root directory, you need to run:
```
docker-compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml up -d
```
If you did any changes to the docker-compose files, make sure to add --build at the end.

To stop the containers, simply run:
```
docker-compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml down
```
You have to install nuctl command line tool to build and deploy serverless functions. Download version 1.5.16. It is important that the version you download matches the version in docker-compose.serverless.yml After downloading the nuclio, give it a proper permission and do a softlink
```
sudo chmod +x nuctl-<version>-linux-amd64
sudo ln -sf $(pwd)/nuctl-<version>-linux-amd64 /usr/local/bin/nuctl
```

Create cvat project inside nuclio dashboard where you will deploy new serverless functions and deploy a couple of DL models. Commands below should be run only after CVAT has been installed using docker-compose because it runs nuclio dashboard which manages all serverless functions.

nuctl create project cvat

nuctl deploy --project-name cvat \
  --path serverless/openvino/dextr/nuclio \
  --volume `pwd`/serverless/common:/opt/nuclio/common \
  --platform local

nuctl deploy --project-name cvat \
  --path serverless/openvino/omz/public/yolo-v3-tf/nuclio \
  --volume `pwd`/serverless/common:/opt/nuclio/common \
  --platform local

Note:

See deploy_cpu.sh for more examples.

GPU Support

You will need to install Nvidia Container Toolkit. Also you will need to add --resource-limit nvidia.com/gpu=1 --triggers '{"myHttpTrigger": {"maxWorkers": 1}}' to the nuclio deployment command. You can increase the maxWorker if you have enough GPU memory. As an example, below will run on the GPU:

nuctl deploy --project-name cvat \
  --path serverless/tensorflow/matterport/mask_rcnn/nuclio \
  --platform local --base-image tensorflow/tensorflow:1.15.5-gpu-py3 \
  --desc "GPU based implementation of Mask RCNN on Python 3, Keras, and TensorFlow." \
  --image cvat/tf.matterport.mask_rcnn_gpu \
  --triggers '{"myHttpTrigger": {"maxWorkers": 1}}' \
  --resource-limit nvidia.com/gpu=1

Note:

The number of GPU deployed functions will be limited to your GPU memory.
See deploy_gpu.sh script for more examples.

Troubleshooting Nuclio Functions:

You can open nuclio dashboard at localhost:8070. Make sure status of your functions are up and running without any error.

Test your deployed DL model as a serverless function. The command below should work on Linux and Mac OS.

image=$(curl https://upload.wikimedia.org/wikipedia/en/7/7d/Lenna_%28test_image%29.png --output - | base64 | tr -d '\n')
cat << EOF > /tmp/input.json
{"image": "$image"}
EOF
cat /tmp/input.json | nuctl invoke openvino.omz.public.yolo-v3-tf -c 'application/json'

20.07.17 12:07:44.519    nuctl.platform.invoker (I) Executing function {"method": "POST", "url": "http://:57308", "headers": {"Content-Type":["application/json"],"X-Nuclio-Log-Level":["info"],"X-Nuclio-Target":["openvino.omz.public.yolo-v3-tf"]}}
20.07.17 12:07:45.275    nuctl.platform.invoker (I) Got response {"status": "200 OK"}
20.07.17 12:07:45.275                     nuctl (I) >>> Start of function logs
20.07.17 12:07:45.275 ino.omz.public.yolo-v3-tf (I) Run yolo-v3-tf model {"worker_id": "0", "time": 1594976864570.9353}
20.07.17 12:07:45.275                     nuctl (I) <<< End of function logs

> Response headers:
Date = Fri, 17 Jul 2020 09:07:45 GMT
Content-Type = application/json
Content-Length = 100
Server = nuclio

> Response body:
[
    {
        "confidence": "0.9992254",
        "label": "person",
        "points": [
            39,
            124,
            408,
            512
        ],
        "type": "rectangle"
    }
]

To check for internal server errors, run docker ps -a to see the list of containers. Find the container that you are interested, e.g., nuclio-nuclio-tf-faster-rcnn-inception-v2-coco-gpu. Then check its logs by docker logs <name of your container> e.g.,
```
docker logs nuclio-nuclio-tf-faster-rcnn-inception-v2-coco-gpu
```
To debug a code inside a container, you can use vscode to attach to a container instructions. To apply your changes, make sure to restart the container.
```
docker restart <name_of_the_container>
```

3 - Mounting cloud storage

AWS S3 bucket as filesystem

Ubuntu 20.04

Mount

Install s3fs:
```
sudo apt install s3fs
```

Enter your credentials in a file ${HOME}/.passwd-s3fs and set owner-only permissions:

echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > ${HOME}/.passwd-s3fs
chmod 600 ${HOME}/.passwd-s3fs

Uncomment user_allow_other in the /etc/fuse.conf file: sudo nano /etc/fuse.conf

Run s3fs, replace bucket_name, mount_point:

s3fs <bucket_name> <mount_point> -o allow_other

For more details see here.

Automatically mount

Follow the first 3 mounting steps above.

Using fstab

Create a bash script named aws_s3_fuse(e.g in /usr/bin, as root) with this content (replace user_name on whose behalf the disk will be mounted, backet_name, mount_point, /path/to/.passwd-s3fs):
```
#!/bin/bash
sudo -u <user_name> s3fs <backet_name> <mount_point> -o passwd_file=/path/to/.passwd-s3fs -o allow_other
exit 0
```
Give it the execution permission:
```
sudo chmod +x /usr/bin/aws_s3_fuse
```

Edit /etc/fstab adding a line like this, replace mount_point):

/absolute/path/to/aws_s3_fuse  <mount_point>     fuse    allow_other,user,_netdev     0       0

Using systemd

Create unit file sudo nano /etc/systemd/system/s3fs.service (replace user_name, bucket_name, mount_point, /path/to/.passwd-s3fs):

[Unit]
Description=FUSE filesystem over AWS S3 bucket
After=network.target

[Service]
Environment="MOUNT_POINT=<mount_point>"
User=<user_name>
Group=<user_name>
ExecStart=s3fs <bucket_name> ${MOUNT_POINT} -o passwd_file=/path/to/.passwd-s3fs -o allow_other
ExecStop=fusermount -u ${MOUNT_POINT}
Restart=always
Type=forking

[Install]
WantedBy=multi-user.target

Update the system configurations, enable unit autorun when the system boots, mount the bucket:

sudo systemctl daemon-reload
sudo systemctl enable s3fs.service
sudo systemctl start s3fs.service

Check

A file /etc/mtab contains records of currently mounted filesystems.

cat /etc/mtab | grep 's3fs'

Unmount filesystem

fusermount -u <mount_point>

If you used systemd to mount a bucket:

sudo systemctl stop s3fs.service
sudo systemctl disable s3fs.service

Microsoft Azure container as filesystem

Ubuntu 20.04

Mount

Set up the Microsoft package repository.(More here)

wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
sudo apt-get update

Install blobfuse and fuse:
```
sudo apt-get install blobfuse fuse
```
For more details see here

Create environments (replace account_name, account_key, mount_point):

export AZURE_STORAGE_ACCOUNT=<account_name>
export AZURE_STORAGE_ACCESS_KEY=<account_key>
MOUNT_POINT=<mount_point>

Create a folder for cache:
```
sudo mkdir -p /mnt/blobfusetmp
```
Make sure the file must be owned by the user who mounts the container:
```
sudo chown <user> /mnt/blobfusetmp
```
Create the mount point, if it doesn’t exists:
```
mkdir -p ${MOUNT_POINT}
```
Uncomment user_allow_other in the /etc/fuse.conf file: sudo nano /etc/fuse.conf

Mount container(replace your_container):

blobfuse ${MOUNT_POINT} --container-name=<your_container> --tmp-path=/mnt/blobfusetmp -o allow_other

Automatically mount

Follow the first 7 mounting steps above.

Using fstab

Create configuration file connection.cfg with same content, change accountName, select one from accountKey or sasToken and replace with your value:

accountName <account-name-here>
# Please provide either an account key or a SAS token, and delete the other line.
accountKey <account-key-here-delete-next-line>
#change authType to specify only 1
sasToken <shared-access-token-here-delete-previous-line>
authType <MSI/SAS/SPN/Key/empty>
containerName <insert-container-name-here>

Create a bash script named azure_fuse(e.g in /usr/bin, as root) with content below (replace user_name on whose behalf the disk will be mounted, mount_point, /path/to/blobfusetmp,/path/to/connection.cfg):
```
#!/bin/bash
sudo -u <user_name> blobfuse <mount_point> --tmp-path=/path/to/blobfusetmp  --config-file=/path/to/connection.cfg -o allow_other
exit 0
```
Give it the execution permission:
```
sudo chmod +x /usr/bin/azure_fuse
```
Edit /etc/fstab with the blobfuse script. Add the following line(replace paths):

/absolute/path/to/azure_fuse </path/to/desired/mountpoint> fuse allow_other,user,_netdev

Using systemd

Create unit file sudo nano /etc/systemd/system/blobfuse.service. (replace user_name, mount_point, container_name,/path/to/connection.cfg):

[Unit]
Description=FUSE filesystem over Azure container
After=network.target

[Service]
Environment="MOUNT_POINT=<mount_point>"
User=<user_name>
Group=<user_name>
ExecStart=blobfuse ${MOUNT_POINT} --container-name=<container_name> --tmp-path=/mnt/blobfusetmp --config-file=/path/to/connection.cfg -o allow_other
ExecStop=fusermount -u ${MOUNT_POINT}
Restart=always
Type=forking

[Install]
WantedBy=multi-user.target

Update the system configurations, enable unit autorun when the system boots, mount the container:
```
sudo systemctl daemon-reload
sudo systemctl enable blobfuse.service
sudo systemctl start blobfuse.service
```
Or for more detail see here

Check

A file /etc/mtab contains records of currently mounted filesystems.

cat /etc/mtab | grep 'blobfuse'

Unmount filesystem

fusermount -u <mount_point>

If you used systemd to mount a container:

sudo systemctl stop blobfuse.service
sudo systemctl disable blobfuse.service

If you have any mounting problems, check out the answers to common problems

Google Drive as filesystem

Ubuntu 20.04

Mount

To mount a google drive as a filesystem in user space(FUSE) you can use google-drive-ocamlfuse To do this follow the instructions below:

Install google-drive-ocamlfuse:

sudo add-apt-repository ppa:alessandro-strada/ppa
sudo apt-get update
sudo apt-get install google-drive-ocamlfuse

Run google-drive-ocamlfuse without parameters:
```
google-drive-ocamlfuse
```
This command will create the default application directory (~/.gdfuse/default), containing the configuration file config (see the wiki page for more details about configuration). And it will start a web browser to obtain authorization to access your Google Drive. This will let you modify default configuration before mounting the filesystem.

Then you can choose a local directory to mount your Google Drive (e.g.: ~/GoogleDrive).
Create the mount point, if it doesn’t exist(replace mount_point):
```
mountpoint="<mount_point>"
mkdir -p $mountpoint
```
Uncomment user_allow_other in the /etc/fuse.conf file: sudo nano /etc/fuse.conf

Mount the filesystem:

google-drive-ocamlfuse -o allow_other $mountpoint

Automatically mount

Follow the first 4 mounting steps above.

Using fstab

Create a bash script named gdfuse(e.g in /usr/bin, as root) with this content (replace user_name on whose behalf the disk will be mounted, label, mount_point):
```
#!/bin/bash
sudo -u <user_name> google-drive-ocamlfuse -o allow_other -label <label> <mount_point>
exit 0
```
Give it the execution permission:
```
sudo chmod +x /usr/bin/gdfuse
```

Edit /etc/fstab adding a line like this, replace mount_point):

/absolute/path/to/gdfuse  <mount_point>     fuse    allow_other,user,_netdev     0       0

For more details see here

Using systemd

Create unit file sudo nano /etc/systemd/system/google-drive-ocamlfuse.service. (replace user_name, label(default label=default), mount_point):

[Unit]
Description=FUSE filesystem over Google Drive
After=network.target

[Service]
Environment="MOUNT_POINT=<mount_point>"
User=<user_name>
Group=<user_name>
ExecStart=google-drive-ocamlfuse -label <label> ${MOUNT_POINT}
ExecStop=fusermount -u ${MOUNT_POINT}
Restart=always
Type=forking

[Install]
WantedBy=multi-user.target

Update the system configurations, enable unit autorun when the system boots, mount the drive:

sudo systemctl daemon-reload
sudo systemctl enable google-drive-ocamlfuse.service
sudo systemctl start google-drive-ocamlfuse.service

For more details see here

Check

A file /etc/mtab contains records of currently mounted filesystems.

cat /etc/mtab | grep 'google-drive-ocamlfuse'

Unmount filesystem

fusermount -u <mount_point>

If you used systemd to mount a drive:

sudo systemctl stop google-drive-ocamlfuse.service
sudo systemctl disable google-drive-ocamlfuse.service

4 - Backup guide

About CVAT data volumes

Docker volumes are used to store all CVAT data:

cvat_db: PostgreSQL database files, used to store information about users, tasks, projects, annotations, etc. Mounted into cvat_db container by /var/lib/postgresql/data path.
cvat_data: used to store uploaded and prepared media data. Mounted into cvat container by /home/django/data path.
cvat_keys: used to store user ssh keys needed for synchronization with a remote Git repository. Mounted into cvat container by /home/django/keys path.
cvat_logs: used to store logs of CVAT backend processes managed by supevisord. Mounted into cvat container by /home/django/logs path.
cvat_events: this is an optional volume that is used only when Analytics component is enabled and is used to store Elasticsearch database files. Mounted into cvat_elasticsearch container by /usr/share/elasticsearch/data path.

How to backup all CVAT data

All CVAT containers should be stopped before backup:

docker-compose stop

Please don’t forget to include all the compose config files that were used in the docker-compose command using the -f parameter.

Backup data:

mkdir backup
docker run --rm --name temp_backup --volumes-from cvat_db -v $(pwd)/backup:/backup ubuntu tar -cjvf /backup/cvat_db.tar.bz2 /var/lib/postgresql/data
docker run --rm --name temp_backup --volumes-from cvat -v $(pwd)/backup:/backup ubuntu tar -cjvf /backup/cvat_data.tar.bz2 /home/django/data
# [optional]
docker run --rm --name temp_backup --volumes-from cvat_elasticsearch -v $(pwd)/backup:/backup ubuntu tar -cjvf /backup/cvat_events.tar.bz2 /usr/share/elasticsearch/data

Make sure the backup archives have been created, the output of ls backup command should look like this:

ls backup
cvat_data.tar.bz2  cvat_db.tar.bz2  cvat_events.tar.bz2

How to restore CVAT from backup

Note: CVAT containers must exist (if no, please follow the installation guide). Stop all CVAT containers:

docker-compose stop

Restore data:

cd <path_to_backup_folder>
docker run --rm --name temp_backup --volumes-from cvat_db -v $(pwd):/backup ubuntu bash -c "cd /var/lib/postgresql/data && tar -xvf /backup/cvat_db.tar.bz2 --strip 4"
docker run --rm --name temp_backup --volumes-from cvat -v $(pwd):/backup ubuntu bash -c "cd /home/django/data && tar -xvf /backup/cvat_data.tar.bz2 --strip 3"
# [optional]
docker run --rm --name temp_backup --volumes-from cvat_elasticsearch -v $(pwd):/backup ubuntu bash -c "cd /usr/share/elasticsearch/data && tar -xvf /backup/cvat_events.tar.bz2 --strip 4"

After that run CVAT as usual:

docker-compose up -d

Additional resources

Docker guide about volume backups