Installation for production use

Distribution structure and the configuration fileDistribution structure and the values.yamlconfiguration file

This script uses standard kubectl and helm (package manager for Kubernetes) commands. All available deployment configuration options are described in the helm/values.yaml file.

Distribution structureDistribution structure

• CHANGELOG.md: List of changes for all release versions.

• EULA.md: End user license agreement. Describes the terms of use of the software.

• README.md: Main documentation for installation and configuration.

• datalens-enterprise-images.txt: List of Docker images and their versions used in the distribution.

• datalens-enterprise-images.tar.gz: Docker image archive.

• init.sh: Main script for distribution deployment and administration.

• meta.json: Current release metadata, including key component versions and release checksums.

• bin/: Contains the executable files of the following utilities:

  • kubectl: Tools for interaction with a Kubernetes cluster.
  • helm: Package manager for Helm chart management.
  • helm-diff: Auxiliary utility for Helm release management; helps display the list of changes during updates.
  • stern: Kubernetes cluster log viewing utility.
  • k9s: Kubernetes cluster administration utility with a visual interface inside the terminal.
  • prgz: Utility that displays the progress of unpacking of archives inside a distribution. Required to maintain the SSH session up and running while unpacking.
  • yq: Tool for processing and parsing of JSON and YAML files.

• helm/: Contains Helm charts and deployment templates:

  • Chart.yaml: Main Helm chart info.
  • values.yaml: Configuration file with default values.
  • templates/: Kubernetes manifest templates (deployment, service, volume, etc.).

• help/: Contains auxiliary instructions and configuration examples:

  • docs/id-sync.md/pdf/html: Guide on how to set up a script to synchronize user groups with external providers.
  • docs/ldap.md/pdf/html: Guide on how to set up integration with LDAP.
  • docs/oidc.md/pdf/html: Guide on how to set up integration with OpenID, compatible with an external authentication provider, using Keycloak as an example.
  • auth-provider-config.example.json: Example configuration for authentication providers.
  • docker-config.example.json: Example configuration for access to a private Docker image registry.
  • idp-sync.js: Example of synchronizing user groups from an external IdP (Identity Provider) system.
  • values.example.yaml: Example of a completed configuration file with correctly escaped values. The values ​​are for illustration only and cannot be used for deployment as shown here.

• k3s/: Contains resources to install a K3s cluster in full offline mode:

  • install.sh: K3s cluster installation and configuration script.
  • k3s: K3s cluster management utility.
  • k3s-images.tar.zst: Archive with system images for a K3s cluster.

Flexible configuration via the fileFlexible configuration via the values.yamlfile

The helm/values.yaml file forms a part of the basic deployment template. If the configuration settings in this file are edited directly, you will get an incorrect distribution checksum during deployment.

When using the --values parameter, all other arguments provided to the init.sh script and related to the distribution deployment configuration will be ignored. The script will consider my-prod-values.yaml to be the single point of truth for deployment; therefore, the file must correctly describe all the deployment parameters:

cp ./helm/values.example.yaml ./my-prod-values.yaml
# ... edit my-prod-values.yaml ...
./init.sh --values ./my-prod-values.yaml

If you need to override only some of the parameters via the my-prod-values.yaml file, the final command should look like this:

cp ./helm/values.example.yaml ./my-prod-values.yaml
# ... edit my-prod-values.yaml ...
./init.sh --values ./my-prod-values.yaml --values-merge

The --values-merge parameter allows you to specify an incomplete values.yaml file as the configuration source. Lacking parameters will be substituted from the default values.

Overview of distribution services and their network interactionsOverview of distribution services and their network interactions

DataLens is a bunch of microservices. The installation script will deploy them to Kubernetes. External traffic from the user will go to the Ingress controller, which will direct it to ui. Next, ui will interact with backend services (us, control_api, data_api) via internal Kubernetes network names.

Deployment with an external PostgreSQL clusterDeployment with an external PostgreSQL cluster

An embedded database is an inefficient practice for a production environment. Use an external managed PostgreSQL cluster instead. With great numbers of users, this will give you flexibility when using the service's objects allow you to build a fault-tolerant architecture.

Algorithm of actionsAlgorithm of actions

1. In the my-prod-values.yaml file, disable the embedded PostgreSQL:

   infra:
     postgres:
       enabled: false # Disable the embedded PostgreSQL
   

2. In the same file, under postgres, specify the parameters for connection to the external cluster:

   postgres:
     POSTGRES_HOST: 'your-pg-host.db.example.com'
     POSTGRES_PORT: '5432'
     # ... as well as other user and database parameters, if they differ from the standard ones
   

3. At the start, you can specify your passwords in init.sh, but this is not safe: it is better to provide them via secrets.

When you run the installation with such values.yaml, DataLens will not create its own PostgreSQL pod but will immediately connect to the external cluster you specify.

Using your own TLS certificatesUsing your own TLS certificates

Self-signed certificates are suitable for testing; however, in production, browsers will warn users to use certificates issued by your corporate or public certificate authority (CA).

This is done through the --ingress-tls-crt and --ingress-tls-key arguments:

./init.sh \
--k3s-install \
--ingress-domain datalens.mycompany.com \
--ingress-tls \
--ingress-tls-crt /path/to/your/cert.pem \
--ingress-tls-key /path/to/your/private.key

The script will save these certificates in Kubernetes secrets, and the Ingress controller will use them to terminate TLS traffic.

Some aspects of deploying the Highcharts library within the corporate networkSome aspects of deploying the Highcharts library within the corporate network

By default, some chart types use the Highcharts library. There are plans to migrate future releases to the D3 library. No additional licensing is required; the user acts as an analyst.

Guidelines for designing a fault-tolerant architectureGuidelines for designing a fault-tolerant architecture

High availability (HA) is achieved by duplicating components. In Kubernetes, you do this by increasing the number of replicas.

The values.yaml file has the application section in which you can specify replicas for each service:

application:
  control_api:
    replicas: 2 # Run two instances of `control-api`, which is a connection and dataset management service
  data_api:
    replicas: 3 # Run three instances of `data-api`, which is a service that accesses the source
  us:
    replicas: 2 # Run two instances of `us`, which is a metadata storage service
  # ... and so forth

Scaling recommendationsScaling recommendations

• Start scaling the most loaded components: data_api, control_api, us.
• For full-value HA, you need a fault-tolerant Kubernetes cluster of multiple nodes (servers). An installation with the --k3s-install flag creates a single-node cluster and is not fault-tolerant.
• Use an external PostgreSQL and Redis® fault-tolerant cluster.

Practical training: Setting up for productionPractical training: Setting up values.yamlfor production

Prepare a configuration file for a production installation scenario.

Goal: Creating a prod-config.yaml file that configures DataLens to work with external PostgreSQL and disables demo data.

StepsSteps

1. Copy ./helm/values.example.yaml to the file named prod-config.yaml.

2. Open prod-config.yaml in a text editor.

3. Find the infra.postgres section and set enabled: false.

4. Under postgres, change POSTGRES_HOST to prod-pg.db.internal.

5. Find the features.demo section and set enabled: false to not deploy demo dashboards to the production environment.

6. Save the file. There is no need to perform the actual installation, just prepare the file.

# ...
infra:
  postgres:
  enabled: false
# ...
postgres:
  POSTGRES_HOST: prod-pg.db.internal
# ...
features:
  demo:
    enabled: false
# ...

ResultsResults

You have covered some important aspects of installing DataLens for production use. Now you know how to manage your configuration using values.yaml and connect external systems.