Skip to content

Grafana Loki

By enabling this toolset, HolmesGPT will fetch pod logs from Loki. Loki can be accessed directly or by proxying through a Grafana instance.

You should enable this toolset to replace the default Kubernetes logs toolset if all your kubernetes/pod logs are consolidated inside Loki. It will make it easier for HolmesGPT to fetch incident logs, including the ability to precisely consult past logs.

Logging Toolsets

Only one logging toolset should be enabled at a time. If you enable this toolset, disable the default kubernetes/logs toolset.

HolmesGPT provides several out-of-the-box alternatives for log access. You can select from these options:

Proxying through Grafana

This is the recommended approach because we intend to add more capabilities to the toolset that are only available with Grafana.

Prerequisites

A Grafana service account token with the following permissions:

  • Basic role -> Viewer
  • Data sources -> Reader

Check out this video on creating a Grafana service account token.

Getting Grafana URL

You can find the Grafana URL required for Loki in your Grafana cloud account settings.

Obtaining the datasource UID

You may have multiple Loki data sources setup in Grafana. HolmesGPT uses a single Loki datasource to fetch the logs and it needs to know the UID of this datasource.

A simple way to get the datasource UID is to access the Grafana API by running the following request:

# port forward if you are using Robusta's grafana from your kubernetes cluster
kubectl port-forward svc/robusta-grafana 3000:80

# List the loki data sources
curl -s -u <username>:<password> http://localhost:3000/api/datasources | jq '.[] | select(.type == "loki")'

This will return something like:

{
    "id": 2,
    "uid": "klja8hsa-8a9c-4b35-1230-7baab22b02ee",
    "orgId": 1,
    "name": "Loki-kubernetes",
    "type": "loki",
    "typeName": "Loki",
    "typeLogoUrl": "/public/app/plugins/datasource/loki/img/loki_icon.svg",
    "access": "proxy",
    "url": "http://loki.loki:3100",
    "user": "",
    "database": "",
    "basicAuth": false,
    "isDefault": false,
    "jsonData": {
        "httpHeaderName1": "admin",
        "httpHeaderName2": "X-Scope-OrgID",
        "tlsSkipVerify": true
    },
    "readOnly": false
}

In this case, the Loki datasource UID is klja8hsa-8a9c-4b35-1230-7baab22b02ee.

Configuration (Grafana Proxy)

Add the following to ~/.holmes/config.yaml, creating the file if it doesn't exist:

toolsets:
  grafana/loki:
    enabled: true
    config:
      api_key: <your grafana API key>
      url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL
      grafana_datasource_uid: <the UID of the loki data source in Grafana>

  kubernetes/logs:
    enabled: false # HolmesGPT's default logging mechanism MUST be disabled

holmes:
  toolsets:
    grafana/loki:
      enabled: true
      config:
        api_key: <your grafana API key>
        url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL
        grafana_datasource_uid: <the UID of the loki data source in Grafana>

    kubernetes/logs:
      enabled: false # HolmesGPT's default logging mechanism MUST be disabled

Update your Helm values and run a Helm upgrade: 

helm upgrade robusta robusta/robusta --values=generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Direct Connection

The toolset can directly connect to a Loki instance without proxying through a Grafana instance. This is done by not setting the grafana_datasource_uid field. Not setting this field makes HolmesGPT assume that it is directly connecting to Loki.

Configuration (Direct Connection)

Add the following to ~/.holmes/config.yaml, creating the file if it doesn't exist:

toolsets:
  grafana/loki:
    enabled: true
    config:
      url: http://loki.logging
      headers:
        X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if loki multitenancy is enabled

  kubernetes/logs:
    enabled: false # HolmesGPT's default logging mechanism MUST be disabled

holmes:
  toolsets:
    grafana/loki:
      enabled: true
      config:
        url: http://loki.logging
        headers:
          X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if loki multitenancy is enabled

    kubernetes/logs:
      enabled: false # HolmesGPT's default logging mechanism MUST be disabled

Update your Helm values and run a Helm upgrade: 

helm upgrade robusta robusta/robusta --values=generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Advanced Configuration

Search Labels

You can tweak the labels used by the toolset to identify kubernetes resources. This is only needed if your Loki logs settings for pod, and namespace differ from the defaults.

Add the following to ~/.holmes/config.yaml:

toolsets:
  grafana/loki:
    enabled: true
    config:
      url: ...
      labels:
          pod: "pod"
          namespace: "namespace"

holmes:
  toolsets:
    grafana/loki:
      enabled: true
      config:
        url: ...
        labels:
            pod: "pod"
            namespace: "namespace"

Use the following commands to list Loki's labels and determine which ones to use:

# Make Loki accessible locally
kubectl port-forward svc/loki 3100:3100

# List all labels. You may have to add the -H 'X-Scope-OrgID:<org id>' option with a valid org id
curl http://localhost:3100/loki/api/v1/labels

Capabilities

Tool Name Description
fetch_pod_logs Fetches pod logs from Loki