Grafana Loki¶

Connect HolmesGPT to Loki for log analysis through Grafana or direct API access. Provides access to historical logs and advanced log queries.

When to Use This¶

✅ Your Kubernetes logs are centralized in Loki
✅ You need historical log data beyond what's in pods
✅ You want advanced log search capabilities

Prerequisites¶

Loki instance with logs from your Kubernetes cluster
Grafana with Loki datasource configured (recommended) OR direct Loki API access

Important: Disable Default Logging Toolset

Only one logging toolset should be enabled at a time. If you enable this toolset, disable the default kubernetes/logs toolset.

Available Log Sources:

Kubernetes logs - Direct pod log access (enabled by default)
Grafana Loki - Centralized logs via Loki
OpenSearch logs - Logs from OpenSearch/Elasticsearch
Coralogix logs - Logs via Coralogix platform
DataDog - Logs from DataDog

💡 Choose one: Only enable one logging toolset at a time for best performance.

Configuration¶

Choose one of the following methods:

Option 1: Through Grafana (Recommended)¶

Required: - Grafana service account token with Viewer role - Loki datasource UID from Grafana

Find your Loki datasource UID:

# Port forward to Grafana
kubectl port-forward svc/grafana 3000:80

# Get Loki datasource UID
curl -s -u admin:admin http://localhost:3000/api/datasources | jq '.[] | select(.type == "loki") | .uid'

Configuration (Grafana Proxy)¶

Holmes CLIRobusta Helm Chart

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

toolsets:
  grafana/loki:
    enabled: true
    config:
      api_key: <your grafana API key>
      url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL
      grafana_datasource_uid: <the UID of the loki data source in Grafana>

  kubernetes/logs:
    enabled: false # HolmesGPT's default logging mechanism MUST be disabled

After making changes to your configuration, run:

holmes toolset refresh

holmes:
  toolsets:
    grafana/loki:
      enabled: true
      config:
        api_key: <your grafana API key>
        url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL
        grafana_datasource_uid: <the UID of the loki data source in Grafana>

    kubernetes/logs:
      enabled: false # HolmesGPT's default logging mechanism MUST be disabled

Update your Helm values and run a Helm upgrade:

helm upgrade robusta robusta/robusta --values=generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Direct Connection¶

The toolset can directly connect to a Loki instance without proxying through a Grafana instance. This is done by not setting the grafana_datasource_uid field. Not setting this field makes HolmesGPT assume that it is directly connecting to Loki.

Configuration (Direct Connection)¶

Holmes CLIRobusta Helm Chart

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

toolsets:
  grafana/loki:
    enabled: true
    config:
      url: http://loki.logging
      headers:
        X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if loki multitenancy is enabled

  kubernetes/logs:
    enabled: false # HolmesGPT's default logging mechanism MUST be disabled

After making changes to your configuration, run:

holmes toolset refresh

holmes:
  toolsets:
    grafana/loki:
      enabled: true
      config:
        url: http://loki.logging
        headers:
          X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if loki multitenancy is enabled

    kubernetes/logs:
      enabled: false # HolmesGPT's default logging mechanism MUST be disabled

Update your Helm values and run a Helm upgrade:

helm upgrade robusta robusta/robusta --values=generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Advanced Configuration¶

Search Labels¶

You can tweak the labels used by the toolset to identify Kubernetes resources. This is only needed if your Loki logs settings for pod and namespace differ from the defaults.

Holmes CLIRobusta Helm Chart

Add the following to ~/.holmes/config.yaml:

toolsets:
  grafana/loki:
    enabled: true
    config:
      url: ...
      labels:
          pod: "pod"
          namespace: "namespace"

holmes:
  toolsets:
    grafana/loki:
      enabled: true
      config:
        url: ...
        labels:
            pod: "pod"
            namespace: "namespace"

Use the following commands to list Loki's labels and determine which ones to use:

# Make Loki accessible locally
kubectl port-forward svc/loki 3100:3100

# List all labels. You may have to add the -H 'X-Scope-OrgID:<org id>' option with a valid org id
curl http://localhost:3100/loki/api/v1/labels

Capabilities¶

Tool Name	Description
fetch_pod_logs	Fetches pod logs from Loki