Accelerate Incident Response with Grafana Assistant's Autonomous Infrastructure Knowledge

By • min read

Overview

When an unexpected alert fires, every second counts. Traditional AI assistants require you to share context about your data sources, services, and dependencies before they can help—wasting precious time. Grafana Assistant changes this by building a persistent knowledge base of your infrastructure before you ask your first question. It automatically discovers your Prometheus, Loki, and Tempo data sources, scans metrics, correlates logs and traces, and generates structured documentation for each service group. This guide walks you through how the assistant works, how to set it up, and how to leverage its pre-loaded context to slash your mean time to resolution (MTTR).

Prerequisites

• Grafana Cloud account – Grafana Assistant is a feature of Grafana Cloud (not self-hosted Grafana). Ensure you have an active stack.
• Connected data sources – At least one Prometheus, Loki, or Tempo data source must be configured in your Grafana Cloud stack. The assistant works best when all three are present.
• Permissions – You need Admin or Editor role in the stack to enable and use Assistant.
• Basic familiarity – Understanding of metrics, logs, and traces (e.g., what a service map is) will help you interpret the assistant's outputs.

Step-by-Step Instructions

1. Enable Grafana Assistant in Your Stack

Navigate to your Grafana Cloud stack's settings page. Under the Observability section, toggle on Grafana Assistant. Once enabled, a swarm of AI agents begins working in the background—no further configuration required. The assistant will automatically detect all Prometheus, Loki, and Tempo data sources already connected to your stack.

2. Data Source Discovery

The first agent performs a data source inventory. It lists every Prometheus, Loki, and Tempo instance in your stack. This happens within minutes of enabling Assistant. You can verify by opening the Assistant panel (? icon → Assistant) and asking "What data sources do you see?"

3. Metrics Scan for Services and Deployments

Separate agents (one per Prometheus data source) run parallel queries to discover:

• Targets and their job labels
• Service names (commonly from service_name or app labels)
• Deployment environments (e.g., environment, namespace)
• Infrastructure components (e.g., container, pod, host)

Example query the assistant might use internally: count by (job, service_name) ({__name__=~"up|process_cpu_seconds_total"}). The agents compile this into a list of unique service groups.

4. Enrichment via Logs and Traces

With the service list in hand, a second wave of agents correlates:

• Log formats – By querying Loki for recent log streams of each service, the assistant identifies whether logs are JSON, plaintext, or structured key-value pairs.
• Trace structures – From Tempo, it examines a sample of traces to understand parent-child span relationships and typical latencies.
• Dependencies – Using trace span attributes like http.target, rpc.service, or db.name, it maps which services call which.

This step enriches the raw metric data with contextual layers—transforming a service list into a true dependency graph.

5. Structured Knowledge Generation

For each discovered service group, the assistant produces a mini documentation file covering five sections:

1. What is this service? – A description inferred from job names, labels, and traces (e.g., "payment-service handles checkout transaction processing").
2. Key metrics & labels – High-cardinality labels, important metrics like http_requests_total, latency_seconds, and any SLO-related metrics.
3. Deployment details – Namespace, cluster, deployment strategy (if detectable from label patterns like deployment= or version).
4. Dependencies – Upstream and downstream services, databases, and message queues identified from trace data.
5. Where to find logs/traces – Specific Loki log streams and Tempo trace queries that best represent the service.

All this is stored in a persistent knowledge base that the assistant can retrieve instantly.

6. Using the Pre-Loaded Context

Once the knowledge base is built (typically within a few hours for a moderate-size stack), you can start troubleshooting without context sharing. Try these example prompts:

• "Why is the payment service slow?" – The assistant already knows its dependencies and can check latency metrics instantly.
• "Show me recent errors in the checkout service." – It directs you to the correct Loki log stream without you specifying the data source.
• "What services depend on the user database?" – It answers from the dependency map, no need to manually trace connections.

The assistant also updates its knowledge base periodically (every few hours) so that new services or changed configurations are reflected automatically.

Common Mistakes

Expecting Instant Results

While the first data source scan starts immediately, building a complete knowledge base (especially for large stacks with multiple data sources) can take up to several hours. Be patient—the assistant is learning. You can check its progress by asking "How much do you know about my infrastructure?"

Relying on Unconventional Label Names

The assistant uses heuristics common across many observability setups. If your labels are highly custom (e.g., mycustomlabel instead of service_name or app), the knowledge generation may be less accurate. Consider standardizing on well-known label names for better results.

Ignoring Data Source Permissions

If a Prometheus or Loki data source requires authentication and your Grafana Cloud stack has not stored the credentials properly, Assistant will skip that source. Verify all data sources show as "Connected" and working in the Data Sources page.

Using Assistant Without Logs or Traces

Assistant works with metrics alone, but its enrichment is significantly less powerful. Without Loki and Tempo, it cannot infer dependencies or log formats. For full benefits, ensure all three pillars are connected.

Not Validating the Knowledge Base

After the initial build, ask a few simple questions (e.g., "List all services you know about") and compare with your actual service list. If anything is missing, check that the relevant data sources are being scanned and that your services have basic labels.

Summary

Grafana Assistant transforms incident response by eliminating the need for context sharing. Its autonomous agents discover data sources, scan metrics, correlate logs and traces, and generate a persistent knowledge base of your services, dependencies, and observability data sources. With this pre-loaded context, you can dive directly into troubleshooting—saving minutes during critical outages. Enable Assistant in your Grafana Cloud stack, wait for the knowledge base to build, and start asking questions immediately. The result: faster fixes with less friction.