---
metadata:
  - name: generator
    content: Diplodoc Platform v5.63.0
alternate:
  - en/cookbook/logging
  - ru/cookbook/logging
  - href: en/cookbook/logging.md
    type: text/markdown
    title: Markdown version
csp:
  - script-src:
      - https://mc.yandex.ru
    img-src:
      - https://mc.yandex.ru
    connect-src:
      - https://mc.yandex.ru
      - wss://mc.yandex.ru
    child-src:
      - 'blob:'
      - https://mc.yandex.ru
    frame-src:
      - 'blob:'
      - https://mc.yandex.ru
    frame-ancestors:
      - 'blob:'
      - https://mc.yandex.ru
canonical: en/cookbook/logging.html
vcsPath: en/cookbook/logging.md
---
# Collecting logs

## Two log types in DataLens On-premises

**Application logs**: Application operational logs. These are technical logs generated by each microservice (`ui`, `data_api`, `us`, etc.) and written to `stdout`/`stderr` of every Kubernetes pod. They contain information about runs, internal operations, errors, and stack traces for debugging and investigating Datalens failures.

**Usage Tracking**: Usage logs. They contain structured events related to user activity, e.g., opening a dashboard, running a query, time of response. These logs are collected by `fluent-bit`, which then sends them to a ClickHouse database. Use these logs to analyze dashboard views, identify heavy queries, and for conduct audits.

## Log collection tools

The `init.sh` script provides several tools for working with logs.

### Collecting logs from all pods

Here is the main command you can use to get a high-level view of what is happening.

```shell
./init.sh --stern . -o extjson > datalens.enterprise.log

# --stern .: Aggregates logs from all pods in the DataLens namespace.
# -o extjson: Output is in structured JSON, which is convenient for parsing. For human-readable output, you can use the raw format.
# > datalens.enterprise.log: Redirects the output to a file for further analysis or sending to support.
```

### Collecting logs from a single pod

If you know which specific service is experiencing issues, e.g., `auth`, you can view its logs only.

```shell
 ./init.sh --stern --tail 100 --no-follow pod/<pod_name>
```

The pod name is always unique, so you need to look it up beforehand. You can do this by running this command:

```shell
./init.sh --kubectl get pods
```

### Collecting cluster events

Sometimes issues occur at the Kubernetes level: a pod cannot start, pull an image, or get resources. In this case, you need to review cluster events, not application logs.

```shell
./init.sh --kubectl get events
```

This command will point out one of these problems: failure when pulling a Docker image (`ImagePullBackoff`), pod crushing and restarting (`CrashLoopBackoff`), or a lack of resources.

## Advanced debugging tools

### `k9s` terminal interface

The distribution has a built-in `k9s` utility for interactive monitoring and cluster management.

```shell
./init.sh --k9s
```

This command launches a text interface directly in your terminal. You can monitor pod statuses, view logs, control resource consumption in real time, switch between namespaces, and manage other parameters.

### Collecting a full report for tech support

There is a single command that automatically collects all diagnostic information into one archive.

```shell
./init.sh --get-debug-info
```

It collects logs, Helm release configuration, cluster events, and other service information.

{% note info %}

When you contact support, they will first request this command’s result.

{% endnote %}

## Setting up Usage Tracking with an external ClickHouse database

In production, it is better to use an external ClickHouse database. The setup involves three steps:

1. **Enabling the function**. Install `features.usage_tracking.enabled: true` in `values.yaml`.

1. **Preparing a table**. In a ClickHouse cluster, run the following DDL query from `Readme.md` or local documentation (`/docs`) to create a table with the correct structure.

  ```sql
  CREATE TABLE $CLICKHOUSE_DB_USAGE_TRACKING.$CLICKHOUSE_TABLE_USAGE_TRACKING
  ON CLUSTER '{cluster}' (
      event_time DateTime64(9),
      event_date Date,
      source_entry_id String,
      dash_id Nullable(String),
      dash_tab_id Nullable(String),
      chart_id Nullable(String),
      chart_kind Nullable(String),
      response_status_code Nullable(UInt64),
      dataset_id Nullable(String),
      user_id Nullable(String),
      request_id Nullable(String),
      query Nullable(String),
      source Nullable(String),
      connection_id Nullable(String),
      dataset_mode Nullable(String),
      username Nullable(String),
      execution_time Int64,
      status Nullable(String),
      error Nullable(String),
      connection_type Nullable(String),
      host Nullable(String),
      cluster Nullable(String),
      clique_alias Nullable(String),
      cache_used UInt8,
      cache_full_hit UInt8,
      endpoint_code Nullable(String),
      query_type Nullable(String),
      err_code Nullable(String),
      workbook_id Nullable(String)
  ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/$CLICKHOUSE_DB_USAGE_TRACKING.$CLICKHOUSE_TABLE_USAGE_TRACKING', '{replica}')
  PARTITION BY toYYYYMM(event_date)
  ORDER BY (toStartOfHour(event_time), connection_id, dash_id, dataset_id, chart_id, user_id, event_time)
  TTL event_date + toIntervalMonth(6)
  SETTINGS index_granularity = 8192, allow_nullable_key = 1;
  ```

1. **Configuring DataLens**. Disable the embedded ClickHouse cluster (`infra.clickhouse.enabled: false`) in `values.yaml` and specify the parameters for connecting to your cluster.

    ```yaml
    clickhouse:
      CLICKHOUSE_HOST: 'your-ch.db.internal'
      # ...
      CLICKHOUSE_DB_USAGE_TRACKING: 'your_db_name'
      CLICKHOUSE_TABLE_USAGE_TRACKING: 'your_table_name'
    ```

## Use case

Users are complaining that the DataLens interface is not loading. You check the pods and see that the `datalens-enterprise-ui-...` pod is in the `ImagePullBackoff` state. Where would you look first to find out the cause?

* `datalens-enterprise-ui` pod logs using `stern`
* Cluster events using `./init.sh --kubectl get events`
* Usage Tracking logs in ClickHouse
* `k9s` interface

{% cut "Find out the answer" %}

* `datalens-enterprise-ui` pod logs using `stern`

  Incorrect. No application logs are generated yet because the container cannot even start.

* Cluster events using `./init.sh --kubectl get events`

  Correct! Cluster events will contain detailed information about the `ImagePullBackoff` error, e.g., `incorrect registry address` or `registry authentication error`.

* Usage Tracking logs in ClickHouse

  Wrong. Usage Tracking logs activity in a running application, but, in the case we described, the application cannot start.

* `k9s` interface

  Feedback: In `k9s`, you will see the `ImagePullBackoff` status, but you still need to check the pod's description or `events` for error details. So a more straightforward solution is to review events.

{% endcut %}

## Hands-on task

Connect to your VM with installed DataLens and run the diagnostic commands.

1. View cluster events: `./init.sh --kubectl get events`. Is there anything suspicious?

1. Get the last ten log lines from `ui`:

   ```bash
   ./init.sh --stern --tail 10 deployment/datalens-enterprise-ui
   ```

1. Run `k9s` (`./init.sh --k9s`) and explore its interface. Try to select a pod and press `l` to view its logs. Exit `k9s` using `:q` or Ctrl + C.

## Summary

Now you have a complete set of tools for DataLens diagnostics: from viewing general logs to analyzing specific cluster events. You know how to quickly collect information for support and how to use interactive monitoring utilities.