Collecting logs

Two log types in DataLens On-premisesTwo log types in DataLens On-premises

Application logs: Application operational logs. These are technical logs generated by each microservice (ui, data_api, us, etc.) and written to stdout/stderr of every Kubernetes pod. They contain information about runs, internal operations, errors, and stack traces for debugging and investigating Datalens failures.

Usage Tracking: Usage logs. They contain structured events related to user activity, e.g., opening a dashboard, running a query, time of response. These logs are collected by fluent-bit, which then sends them to a ClickHouse database. Use these logs to analyze dashboard views, identify heavy queries, and for conduct audits.

Log collection toolsLog collection tools

The init.sh script provides several tools for working with logs.

Collecting logs from all podsCollecting logs from all pods

Here is the main command you can use to get a high-level view of what is happening.

./init.sh --stern . -o extjson > datalens.enterprise.log

# --stern .: Aggregates logs from all pods in the DataLens namespace.
# -o extjson: Output is in structured JSON, which is convenient for parsing. For human-readable output, you can use the raw format.
# > datalens.enterprise.log: Redirects the output to a file for further analysis or sending to support.

Collecting logs from a single podCollecting logs from a single pod

If you know which specific service is experiencing issues, e.g., auth, you can view its logs only.

 ./init.sh --stern --tail 100 --no-follow pod/<pod_name>

The pod name is always unique, so you need to look it up beforehand. You can do this by running this command:

./init.sh --kubectl get pods

Collecting cluster eventsCollecting cluster events

Sometimes issues occur at the Kubernetes level: a pod cannot start, pull an image, or get resources. In this case, you need to review cluster events, not application logs.

./init.sh --kubectl get events

This command will point out one of these problems: failure when pulling a Docker image (ImagePullBackoff), pod crushing and restarting (CrashLoopBackoff), or a lack of resources.

Advanced debugging toolsAdvanced debugging tools

terminal interfacek9s terminal interface

The distribution has a built-in k9s utility for interactive monitoring and cluster management.

./init.sh --k9s

This command launches a text interface directly in your terminal. You can monitor pod statuses, view logs, control resource consumption in real time, switch between namespaces, and manage other parameters.

Collecting a full report for tech supportCollecting a full report for tech support

There is a single command that automatically collects all diagnostic information into one archive.

./init.sh --get-debug-info

It collects logs, Helm release configuration, cluster events, and other service information.

Setting up Usage Tracking with an external ClickHouse databaseSetting up Usage Tracking with an external ClickHouse database

In production, it is better to use an external ClickHouse database. The setup involves three steps:

1. Enabling the function. Install features.usage_tracking.enabled: true in values.yaml.

2. Preparing a table. In a ClickHouse cluster, run the following DDL query from Readme.md or local documentation (/docs) to create a table with the correct structure.

CREATE TABLE $CLICKHOUSE_DB_USAGE_TRACKING.$CLICKHOUSE_TABLE_USAGE_TRACKING
ON CLUSTER '{cluster}' (
    event_time DateTime64(9),
    event_date Date,
    source_entry_id String,
    dash_id Nullable(String),
    dash_tab_id Nullable(String),
    chart_id Nullable(String),
    chart_kind Nullable(String),
    response_status_code Nullable(UInt64),
    dataset_id Nullable(String),
    user_id Nullable(String),
    request_id Nullable(String),
    query Nullable(String),
    source Nullable(String),
    connection_id Nullable(String),
    dataset_mode Nullable(String),
    username Nullable(String),
    execution_time Int64,
    status Nullable(String),
    error Nullable(String),
    connection_type Nullable(String),
    host Nullable(String),
    cluster Nullable(String),
    clique_alias Nullable(String),
    cache_used UInt8,
    cache_full_hit UInt8,
    endpoint_code Nullable(String),
    query_type Nullable(String),
    err_code Nullable(String),
    workbook_id Nullable(String)
) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/$CLICKHOUSE_DB_USAGE_TRACKING.$CLICKHOUSE_TABLE_USAGE_TRACKING', '{replica}')
PARTITION BY toYYYYMM(event_date)
ORDER BY (toStartOfHour(event_time), connection_id, dash_id, dataset_id, chart_id, user_id, event_time)
TTL event_date + toIntervalMonth(6)
SETTINGS index_granularity = 8192, allow_nullable_key = 1;

1. Configuring DataLens. Disable the embedded ClickHouse cluster (infra.clickhouse.enabled: false) in values.yaml and specify the parameters for connecting to your cluster.

   clickhouse:
     CLICKHOUSE_HOST: 'your-ch.db.internal'
     # ...
     CLICKHOUSE_DB_USAGE_TRACKING: 'your_db_name'
     CLICKHOUSE_TABLE_USAGE_TRACKING: 'your_table_name'
   

Use caseUse case

Users are complaining that the DataLens interface is not loading. You check the pods and see that the datalens-enterprise-ui-... pod is in the ImagePullBackoff state. Where would you look first to find out the cause?

• datalens-enterprise-ui pod logs using stern
• Cluster events using ./init.sh --kubectl get events
• Usage Tracking logs in ClickHouse
• k9s interface

Hands-on taskHands-on task

Connect to your VM with installed DataLens and run the diagnostic commands.

1. View cluster events: ./init.sh --kubectl get events. Is there anything suspicious?

2. Get the last ten log lines from ui:

   ./init.sh --stern --tail 10 deployment/datalens-enterprise-ui
   

3. Run k9s (./init.sh --k9s) and explore its interface. Try to select a pod and press l to view its logs. Exit k9s using :q or Ctrl + C.

SummarySummary

Now you have a complete set of tools for DataLens diagnostics: from viewing general logs to analyzing specific cluster events. You know how to quickly collect information for support and how to use interactive monitoring utilities.