Migration and backup

Two approaches to data migration

  1. Backup and recovery (full)

    Creating a full copy of metadata of the entire system. Used for Disaster Recovery or to clone an environment (e.g., dev → test).

  2. Export and import (partial)

    This is a transfer of individual objects (workbooks with their contents). Used for transfer of finished reports between environments (e.g., test → prod) or for migration from a cloud.

Backup and recovery of PostgreSQL databases

The whole DataLens configuration (users, dashboards, permissions) is stored in a PostgreSQL service database. A backup of this database is a backup of the entire system. The init.sh script provides convenient commands for this:

  • To create a backup

    ./init.sh --dump-postgres > datalens_backup.sql

    This command will create an SQL dump of the database and output it to stdout, which will be forwarded to a file. Running this command regularly is the foundation of your backup strategy.

  • To recover from a backup

    ./init.sh --restore-postgres datalens_backup.sql

    This command will deploy the data from the SQL file you specify into a database.

    Warning

    The data in the database will be overwritten.

Why CONTROL_API_CRYPTO_KEY is important

There is a critical nuance when recovering from a backup on a new DataLens installation. Passwords and tokens in connections are encrypted with CONTROL_API_CRYPTO_KEY. If this key is different on the new system, DataLens will not be able to decrypt the old secrets, and all connections will stop working.

Tip

When migrating via backup and recovery, it is important to transfer not only the database dump, but also the CONTROL_API_CRYPTO_KEY value from the Kubernetes secret of the old system to the new one.

Exporting and importing workbooks

This mechanism is designed to transfer specific content. For it to work, the export feature must be enabled on both installations.

Process:

  1. Source system

    User with permissions for the workbook selects Export. DataLens creates a ZIP archive with all objects inside the workbook.

  2. Target system

    User with the administrator or editor role selects Import and uploads the resulting archive.

After importing, you may need to link datasets to new connections, because connection IDs may not be the same on different systems.

Migration scenario: Yandex Cloud → On-premises

This is a frequent task for a company to migrate its analytics into its corporate network. As there is no direct access to the cloud DataLens' service database, the export and import method is used.

Migration plan:

  1. Preparation

    Deploy and configure DataLens On-premises, including authorization and network access to data sources.

  2. Creating connections

    In the new On-premises environment, manually create all connections to the data sources that were used in the cloud.

  3. Export

    In the cloud DataLens, export all required workbooks into JSON files in succession.

  4. Import

    In the On-premises environment, import the resulting JSON files.

  5. Rebinding

    After importing, open the datasets and make sure they are using the new connections.

Use case

You are migrating DataLens to a new server using pg_dump/pg_restore. Following a recovery, all dashboards are in place, but the charts do not work and return the Connection failed error. What did you forget to do?

  • Enable the export feature.

  • Transfer CONTROL_API_CRYPTO_KEY from the old server to the new one.

  • Run ./init.sh --get-admin-password.

  • Recreate all connections manually.

Find out the answer

  • Enable the export feature.

    The export feature is needed for transferring individual workbooks, not for recovering from a full backup.

  • Transfer CONTROL_API_CRYPTO_KEY from the old server to the new one.

    Correct! Without the old key, DataLens cannot decrypt the stored database access credentials, which leads to connection errors.

  • Run ./init.sh --get-admin-password.

    This command only displays the administrator password and does not affect the connections.

  • Recreate all connections manually.

    This would be a workaround, but the problem is rooted elsewhere. The correct solution is to move the key.

Practical training: Creating and recovering a backup

Simulate the Disaster Recovery process on your bench.

  1. Create a backup of the current state:

    ./init.sh --dump-postgres > my_first_backup.sql

    A file named my_first_backup.sql will appear.

  2. Navigate to the DataLens interface and delete some object.

  3. Make sure the object is deleted for real.

  4. Recover the system from a backup:

    ./init.sh --restore-postgres my_first_backup.sql

  5. After the command is completed, refresh the page in your browser. The deleted object should reappear.

Summary

You have mastered critical skills: backup, recovery, and migration. Now you know how to deploy and configure DataLens On-premises, and maintain its stable operation over the long term.

You are ready for the real challenges of administering this powerful BI platform. Good luck in your work!

Previous
Connecting specific features