Upgrade Varfish Installation

This section contains upgrade instructions for upgrading your VarFish Server installation using VarFish Docker Compose.

Problem with Data Release 20210728 and GRCh37

The data release has a problem with the GRCh37 extra annotations. If you can then use the updated site data 20210728b release. If you already have an instance with 20210728 background data then you can use the following data file.

Download and extract the file and mount it as /data inside the varfish-web container. You can then apply the patch to your database with the following command.

$ docker exec -it varfish-docker-compose_varfish-web_1 python /usr/src/app/manage.py \
    import_tables --tables-path /data --truncate --force

You can find out more details, give feedback, and ask for help in this Github discussion.

v1.2.* to v2.*.*

ClinVar Changes Please follow the instructions described in v1.2.2 to v1.2.3 if you start at v1.2.1.

In-House Background Database. A number of changes were made to the implementation of the background database. The upgrade will re-create the in-house database as empty.

You will have to re-build the database manually with the command python manage.py rebuild_variant_summary. Assuming that you are running within varfish-docker-compose, you can use the following command directly.

$ docker exec -it varfish-docker-compose_varfish-web_1 python /usr/src/app/manage.py \
    rebuild_variant_summary

Structural Variants. In case that the support for structural variants has been used, it is strongly recommended to re-annotate the structural variants with an updated version of varfish-annotator (v0.24 or above). You will need to use varfish-cli in a recent version (v0.3.4 or above) for being able to import the data into VarFish. Otherwise, see below for background and explanation on how to fill empty values in your database after upgrading the software an database to bollonaster.

In VarFish anthenea, the support for structural variants has been experimental only. To prepare for improved support of structural variants, the database structure has been changed. Prior, the number of carriers in the background had to be annotated outside of VarFish (in your pipeline). This has changed now and VarFish can build a background database (and will do so weekly on sundays).

If you do not want to wait for next sunday to get your background database, you can force building a new background structural variant data set with:

$ docker exec -it varfish-docker-compose_varfish-web_1 python /usr/src/app/manage.py \
    svs_bg_sv_set_build

New fields have been added for the proper representation of break-ends in VarFish. They will only be properly filled after re-import of re-annotated data as described above. You can fill the fields with reasonable values (that will work well for all cases except for breakends/BNDs where the connectivity of 3’ and 5’ ends cannot be properly detected without re-annotation) by using the following.

$ docker exec -it varfish-docker-compose_varfish-web_1 python /usr/src/app/manage.py \
    svs_sv_fill_nulls

This is not strictly necessary and it is recommended to re-annotate and re-import.

v1.2.2 to v1.2.3

Downtime Notice The upgrade will require you to rebuild your in-house database. Depending on the number of cases and the performance of your server, this may take a long time. Please plan your update accordingly.

ClinVar Updates First, make sure that you have upgraded the data to 20210728b following Problem with Data Release 20210728 and GRCh37. Then, upgrade by just updating your varfish-docker-compose repository clone and calling docker-compose down && docker-compose up -d.

Next, patch to data version 20210728c using the following instructions.

We have made a larger change to the ClinVar database. You will have to re-import the ClinVar database after upgrade follows.

Download the appropriate data patch from our file server:

GRCh37

varfish-server-background-db-20210728c-grch37.tar.gz

GRCh38

varfish-server-background-db-20210728c-grch38.tar.gz

Extract the output to a folder on your VarFish server, e.g., /data/varfish-data/varfish-server-background-db-20210728c-grch37, such that this folder contains a file import_versions.tsv. Next, edit the docker-compose.yml file of your varfish-docker-compose such that the varfish-web entry’s volumes field reads as follows.

volumes:
  - "/data:/data:ro"

Then, restart VarFish by calling docker-compose down && docker-compose up -d. After startup, you can now do the following if you use GRCh37:

docker exec -it varfish-docker-compose_varfish-web_1 python /usr/src/app/manage.py \
    import_tables --force --truncate --tables-path /data/varfish-server-background-db-20210728c-grch37 \
    --threads=0

If you use GRCh38, use

docker exec -it varfish-docker-compose_varfish-web_1 python /usr/src/app/manage.py \
    import_tables --force --truncate --tables-path /data/varfish-server-background-db-20210728c-grch38 \
    --threads=0

This will import the ClinVar version from the 20210728 release in the fixed format compatible with v1.2.2. Note that this will also import a patch to the TAD data in VarFish.

After importing the data, the precomputed in-house database will have been cleared. This leads to querying being broken. To fix this, you have to run the following.

docker exec -it varfish-docker-compose_varfish-web_1 python /usr/src/app/manage.py \
    rebuild_variant_summary

This command may run for a long time, depending on the number of samples in your database and your disk speed.

In case of any issues, contact us in the Github Discussion or directly by email.

v0.23.0 to v1.2.2

This includes all version in between, v0.23.1, …, v1.2.2.

Summary

This are minor bug fix releases and small added features. You should be able to upgrade by just updating your varfish-docker-compose repository clone and calling docker-compose up -d.

v0.23.1 to v0.23.2

Summary

This is a minor bug fix release that improved the deployment of the VarFish Demo and Kiosk sites. You should be able to upgrade by just updating your varfish-docker-compose repository clone and calling docker-compose up -d.

v0.22.1 to v0.23.0

Summary

  • The Docker Compose installer now provides support for setting up CADD score annotation via cadd-rest-api.

  • The environment variable FIELD_ENCRYPTION_KEY should be setup properly by the user.

  • Two new celery queues are needed: maintenance and export.

  • To enable the new and optional feature for uploading variants to SPANR you have to set the environment variable VARFISH_ENABLE_SPANR_SUBMISSION to 1.

Detailed Instructions

Docker Compose: cadd-rest-api

Update your varfish-docker-compose installation with the changes from the Github repository without installing cadd-rest-api. This will give you commented out lines for running one cadd-rest-api-server and multiple cadd-rest-api-celeryd-worker-? containers. For enabling them, follow the instructions in Install Scoring with CADD.

Additional Celery Queues

After updating your varfish-docker-compose.yml file, ensure that you the two additional containers varfish-celeryd-maintenance and varfish-celeryd-export. These will run the background jobs for running maintenance tasks and export results. They will be started when running docker-compose up.

Environment Variable: FIELD_ENCRYPTION_KEY

Set the environment variable in the .env file as documented in Miscellaneous Configuration. The default value is also stored in the public repository and thus not very secure.